KAREEN KOUDDOUS: All right. Let's get started.
Hi, I'm Kareem Kouddous. I'm currently CTO
and founder at Crowdtap. I'm also a side project addict.
You can find me on GitHub and Twitter at kareemk.
So Ruby is cute. And what I mean by
that is, you know, it's optimized for developer joy.
I mean, anyone who attended Matz's keynote, you know,
can, can feel that. I personally wanted to go
up and hug him after the keynote. It's so
clear that, you know, the genesis of Ruby came
from a place of wanting to create joy in
what you do every day. Happiness in web development.
You know, and Rails, arguably could be cuter. You
know, Rails took those same principles of really loving
what you do every day as a web developer
or software engineer and taking that to the web
stack. And, I'm sure all of you remember the,
you know, blog in fifteen minutes. And that was
magical, because it was so painful before to get
a web application up and running.
And Rails changed everything.
But, I'm sure, as a lot of you also
know, over time as your team grows and you
start to actually build very interesting features, you, you
tend to, things start to get difficult. You know,
tests start to take a long time. You know,
there's great testing ethos in the Ruby community, but,
you know, you have a lot of tests, you
build a lot of features and things start to
get slow.
And what does that do, you know, instead of
being able to push out code into your staging
or production environment immediately, you have to wait, you
know, minutes, maybe even hours, before that gets in
there and you lose flow. You have to context
switch. That's horrible. You know. Your fat models become
obese.
You know, suddenly, you know, you have six engineers
touching your user model and people are repeating the
code all over the place, because they're trying to
do, you know, something that's the same but slightly
different. And, you know, instead of trying to refactor
the app, you know, you want to push features
out. They're duplicating code. And then these things become
really hard to work with and less joyful.
You know, you start to use a lot of
gems, and that's great. You know, that's one of
the wonderful things about the Ruby community is there's
a lot of us and we've contributed so much
to it. But then you, your app starts to
take a long time and, you know, you're building
an interesting application so it has a lot of
features and you have a lot of gems and
that takes a long time. And you stop experimenting.
You stop just putting up console and trying stuff.
That joy of, that first experience with Rails or
Ruby, you know, starts to erode. And then hugs
turn into yelling, right. You, you're in the, you
know, you're in your chat room, just trying to
figure out what's going on, and yelling at each
other. It's, it's, it's no longer joyful.
And fundamentally what that means is you've lost your
agility. And I think that's what Matz was really
trying to say is, agility is about happiness, right.
If you're happy and free to do what you
want with your code, you're gonna take, you're gonna
innovate. You're gonna do interesting things. You're gonna be
creative.
And, as your app, though, this, this starts to
change, and you lose your agility, and we're back
to, you know, waterfall, but, you know, same pain
points, but just manifested in a different way. So.
So, you know, what was really cute, you know,
suddenly is this really scary, ugly thing that has
appeared in your life, and you don't know what
to do with it. You know, you, you want
to figure out a way to solve this, and
so you need to figure out what the root
cause is, right. That's how you solve problems.
So you need to figure out the root cause
and solve this problem. You want this guy out
of your life.
So to illustrate that, I'm gonna go through how
this manifested itself, this problem, this beast in, at
Crowdtap, and, and walk through that to show how
we got to where we are now. And, I'm
gonna start by just giving you a quick overview
of what Crowdtap is. You know, it's, we're the
leading social influence marketing platform. You know we, we
ally with big brands, collaborate with their customers which
inspires advocacy and drives unmatched social activity.
And what all this marketing speak boils down to
is we create awesome tasks, engaging fun tasks, for
the brand's launch that users complete to help brands
market. And those tasks span the full marketing life
cycle. Everything from completing a poll to taking a
photo of doing something fun and sharing that on
Facebook, to getting free product from the brand, sharing
that with friends, throwing a party around it. And
we're constantly adding different tasks, engaging, fun, engaging tasks,
so that the brands can collaborate with their consumers
to market, instead of just shouting at them.
And that's what our vision is for, for Crowdtap.
So let's, let's take a look at this and
you know, a screenshot. And this is one of
our more popular actions, which is a poll. Brands
can get real time feedback on whatever they want.
And here they're asking, specifically, you know, what Cable
networks do, does, do the users currently watch, or
receive.
And the really powerful thing with Crowdtap is we
take all this data that a user, that, that
users generate from answering all these questions and participating
with the brand, and we give the brands very
rich and deep targeting ability, so that not only
can they target based on all the demographic data
on the user: their age, income, et cetera, which
we collect. We, they can also target based on
everything that user has done. The entire history of
what that user has done with, with that brand.
And they can also target based on what they
haven't done. So, it's a very, very rich targeting
engine. But as you can imagine, this becomes a
performance bottleneck very quickly, because on every page load,
we need to know which tasks are available for
a user, and, if targeting, we need to -
you can target based on anything that user has
done. You have to load up the entire history
of that user.
And so, to be clear, this is a really
big performance bottleneck. And this was the first painpoint
that we encountered. Or, or, a major painpoint that
we encountered, along with everything I was discussing. So
we need, you know, we needed to solve this.
And the way to solve this is, you know,
we need to tailor our data structure for the
problem at hand. So, what I mean by that
is, we're generating data, everything that that user is
doing, and we're collecting it in a way from
our main application that makes sense, you know, when
you're doing just basic CRUD. But then we want
to take that data and we want to have
the freedom to denormalize the data. Maybe put in
caching. Maybe use a different database technology.
We want the freedom to experiment and explore with
this data to be able to solve the problem.
And at the same time, we want to do
it in a way that takes us back to
the joy. Back to the little cute, fuzzy animals
we had in our lives and not these beasts,
right.
So how do you do this? So, you know,
the first thing we thought was, OK, we've been
reading, people have blogged about the services. What about
just extracting and, you know, this functionality in services?
But that doesn't work, right. Because even if you
extract your functionality into services, what you really care
about is your models. Right, that's your data.
So if you, there, we're not talking about extracting
business logic, you know, into, you know, single purpose
files. We're, we're talking about being able to work
with our data in a flexible way. And so
extracting it to services doesn't really help.
And, you know, Rails, what about a Rails engine
or gem extraction. Essentially this is all the same
thing. It's just moving furniture around. What you really
want to do is be able to mutate your
data. You want to be able to play with
your, your data structure, and have freedom there.
And this brought us to, you know, what about
app extraction? Why don't we extract this into a
separate app? You know, we can go back to
the, the little cute animal because it's a smaller
app and solve all of those other painpoints. But
at the same time, have, you know, some freedom
to solve the performance problem which was a very
big business problem for us.
And so the first thing we considered is, how
about extracting the app and sharing the database? Well,
this def, this doesn't work, right. Because all, everything
you need to do with your data is in
your main app. Your main app holds that model.
And so if you're gonna do anything interesting with
your data, you're gonna have to do it in
your main app, so what have you done?
Your targeting app isn't, the splitting up of the
app is not solving anything, right? OK. So let's
do this. Let's to synchronous API. And this is,
this is probably what most people think of as
SOA, right. SOA is some RPC mechanism between apps.
But, again, this does not solve this problem, because,
if you're gonna do anything interesting with your data
and you want to mutate your data, you're gonna
have to do that in your main app.
And so you're, you're adding complexity to your main
app. We want to remove complexity from the main
app and put it somewhere else, right, and, and
deal with it somewhere else. So this doesn't, this
doesn't solve the problem. This just adds more layers.
So I hope, by now, it's clear what the
right way to do this is. And for us,
it was.
You, we need to have a separate database for
this, for our targeting application, and we need to
replicate data from our main application into this separate
app, and have the freedom in our main application
to mutate the data as we wish, experiment with
different databases, really go back to being agile in
solving, in solving the problem.
And that's what we did. And it's at that
point that the heavens parted and Nico fell into
our laps. And, and helped us take this architecture
to the next level. So I'll just, just pass
this onto Nico.
NICOLAS VIENNOT: Hello. How are you? My name is
Nico. I'm a PhD student at Columbia University in
the system department. I love, I love opensource, and
you can find me on Twitter and GitHub at
nviennot.
So last year, I came at Crowdtap, and they
just extracted that targeting logic to its own separate
app that we call Sniper. So that was a
great win, because we were able to finally see
the big picture of this application, of that logic,
and that, and so when we extracted that it
was like ten, I mean, five thousand, five thousand
lines of code. And since we, we were on
that app, we could really refactor it quickly because
the tests were running in, like, you know, thirty
seconds.
And, and we, we got to a point where
the app was just less than a thousand lines
of code, because now, because it, it, it got
really intuitive - oh, look, we should put our
data structure like this. What if, what if we
generalize this way around and all? Which we couldn't
do in the main application because everything was kind
of coupled together and it was really hard to
see, you know, the big picture of things.
So, performance went really well after that. We went
from, you know, a second or two to fifty
milliseconds. That was like, yay! Big win!
But, we came to another problem, which was data
replication. And it's actually hard. So, so what we
did first was something pretty naive. So, on the
main application, the publisher's side, we're like, well, let's
just add some callbacks and post the, the data
to Sniper. And then Sniper, the subscribers can be
like, oh, I can take that data and save
it to its database.
But things started to break down in production. We're
like, it's funny, these two databases don't have the
same user count. It's weird. Because, so, we realized,
wait, it's in a different system. Things fail all
the time.
So, for example, if your web server, right after
writing its data to the database, dies, and is
not able to the callback, then, oops, Sniper doesn't
know about it. What if Sniper is down? Well,
it won't receive some updates for five minutes.
So that's a problem.
So we're like, OK, so this is, this is
a problem. And, and at the same time, we're
looking at this. We're like, oh, that's funny. That
looks really like an observer, like in the Rails
sense, right, like, the Sni- the subscriber is really
like observing the data that's changing on the other
app. And we're like, well, that's, that's a cool
concept. What can it do, then? And what we
really want to do with that is to have,
to split our app in a lot of different
services, right.
Like, if we could have this remote observer and,
and rip the data around, we're like, well, why
don't we apply the same thoughts to the email
service. But we couldn't just add callbacks and see
the production system kind of like blowing away like
this.
Which is why we introduced Promiscuous. Promiscuous is a
transparent application level causally consistent data replication framework for
service-oriented architecture. And, I'm gonna explain to you what
it all means.
So Promiscuous is essentially the glue between all of
your services. So you can think of it as
a message bus, where you're gonna have services publishing
their data, and then you're gonna have subscriber apps
that are like, oh, I'm intruding that model and
this one and that, and it's gonna do something
interesting.
And Promiscuous sits at, on your models, between the
models and the database layer. So everything that the
publisher would do, it's gonna propagate those changes down
to the subscriber models that have a view to
anything.
So that's Promiscuous in a nutshell.
And, and because we are really trying to get
back that joy and the simplicity of Rails and
Ruby, Promiscuous had to, like, to, to provide those
three key features.
So first, it tackles data replication in a novel
way, which scales and is actually consistent. It provides
a very simple API, because as a developer, like,
we just want to write one line of code
and be done with it. And, of course, since
we're Rubyists, we want a good testing framework.
So I'm gonna go through all these three points.
So data replication. It's pretty hard to, to get
your head around this. But let's go through an
example. So say you have some sort of Facebook
app where you have users and they can send
messages to each other.
And, and so here, like, say I want to
implement an email service that sends an email to
whoever receive a message, right. Simple. So here, if,
if you would to subscribe to the data that
is interesting, you would see, oh, our user just
signed up. So create user.
And then, that guy just sent a message to
another guy. OK. Create message, great. So when you
create message, you're gonna be like, oh, fetch from
my database. Oh, OK, I have that user, I
can send random a email. OK, hi, whoever, oh
hi and all that.
But since we need to scale, because, you know,
we have a lot of users, we're gonna add
some more curves. So that's what's gonna happen. On
your email service, we can have, say, two workers,
and now, since we need to process this message
in parallel, what's gonna happen? Well, if worker one
is a bit slow sometimes, maybe it's not gonna
process that create user quick enough to put in
database, and so the worker two is gonna be
like, oh, who is, who is that user? Oh,
well I don't find it in my database. Boom
exception. So that's a problem.
And that's, that's a consistency problem.
But promiscuous, it is going to solve this problem
for you, under, transparently, without you having to do
anything. And so it's going to try to paralyze
as much as, messages that it can, but still
serialize the messages that are important.
For example, the create user happens before the create
message. And that's what we mean by causal consistency.
So as in cause and effect.
So let's see how that kind of works. So
why, how is the, is doing this transparently?
So when you send, so when you have a
user that wants to send a message to another
user, it's gonna hit your controller in some ways.
In your controller, what you're gonna do, you're gonna
interact with your models. So what you gonna do?
You're gonna fetch, first of all the current user,
from his cookie, and then you're gonna do user
find on his friend, and then you're gonna finally
create the message with the frame keys that go
by.
And, since Promiscuous sits at your model, on your
models, it can be like, oh, that's what you
did. So he going to infer that the message
create depends on those two piece of data. And
that's how it's able to serialize the messages that
are important and paralyze the ones that are not.
So you could say, well, that's a bit overkill.
Because why don't I just retry until, you know,
the user is in my database because it would
have an exception, right? But it's, it's actually moreso
than that. SO let's take another example to illustrate
this.
SO say you're on Facebook and you have people
posting pictures and you want to provide an email
service that send an email of the picture to
your, all of your friends. So say you have
a user, consider that scenario where you have a
user coming back from spring break, and he deletes
his parents from his friends list and then he
posts a compromising picture, K.
So what happens if you reverse the processing of
the two messages? Well you're gonna send the picture
before having to remove the parents. And you're not
gonna have an exception, you're gonna have a funk
up.
So, so this is how, so, Promiscuous has you
covered. And you don't want to, when you're developing
event-driven services, you, you don't want to, to think
about all the races you could have and all
that. Like, it's way too complicated. But we still
want a very simple API. So.
So essentially, Promiscuous is, is your main databus, right.
And it does, so, so under the cover you
have, you have RabiitMQ which is, which is a
message broker, so you're gonna have publisher applications that
are gonna send their messages on RabbitMQ, and then
you're gonna have subscriber applications that are gonna connect
to RabbitMQ and each of the services can have
their own queue, I think, going to, to receive
those messages through RabbitMQ.
So no subscriber actually talk to the publisher app
directly. So you're gonna have some good decoupling mechanism.
And because we need to synchronize all the workers
together, we use Reddis. So for example, that worker's
like, oh wait, so before I can create that
message I need for you to wait to create
the user, OK. I'm gonna wait for you a
little bit. Oh, you died. Well, in that case
I'm going to process that message. So all of
that is done through Reddis.
So let's, let's see a bit of the API
of Promiscuous. So on the top, I have a
publisher application running with ActiveRecord on PosGres, for example,
and on the bottom I have a subscriber application
running on MongoDB.
So all I need to do as a developer
is to include the Promiscuous publisher mixin and publish
the field that I'm, I'm interested in, for example,
email and name.
On the subscriber's side, I'm going to include the
Promiscuous subscriber mixin and, and, in my subscribe block,
I'm going to put the fields I'm interested in,
too. And that's it.
That's all you need to do. That's the entire
API. That's it.
So, of course, once you start developing this, you
want to start testing that. And Promiscuous provides you
with a testing framework. So, for us, we leave
that interpretation tests are extremely important, and we, we
have to, to test our subscribers in a way
that is decoupled from the publishers.
So the publishers actually, so Promiscuous allows publishers to,
to create a gem to publish a gem with
factories that represent the kind of data that, if
you were to subscribe to their model, you, the
kind of data that you would get.
So to illustrate this, I'm going to show you,
an, a real test of Sniper, and you can
see, on the top, like, we just do create
Crowdtap member instead of the regular Crowdtap create member.
And so by doing this, Promiscuous is going to
take the factory of Crowdtap and, and package the
payload that you have received for real in production
and send it through your, your cloud client. And
so you're gonna be able to test a really
well your application as if it was in production.
So to really demonstrate how easy it is to
use Promiscuous, Kareem is going to show you a
live demonstration.
K.K.: Hello. Thanks Nico. All right. Let's, let's show
you guys how awesome this is.
OK, so what I'm gonna do is I'm gonna
show, show you guys adding, taking one of our
existing models on our, on, on one of our
apps, and subscribing to it in another. So on
the, on the left side, we've got Crowdtap. And
I've got the, one of our models that we're
actually not publishing yet, which is Package, and so
we're gonna take this and we're going to publish
this model, and then, on the, on the other
side, which is Sniper on the right side, which
is the targeting app we were discussing, I'm going
to subscribe to that model so that any changes
to the publisher app are reflected to the, to,
on the subscriber app. And I'm also gonna put
some callbacks in there to bind to any changes
that are happening.
So let's, let's go ahead and do that. So,
all you need to do first is just include,
you know, the Promiscuous publisher, which Nico showed you
in that API example. And then just wrap, you
know, whatever fields you want to publish in a
publish block. And so this case, you don't need
to publish everything. And, and that's important, you want
to figure out what's your public, what's your API,
right, and doesn't need to be the entire model.
So you publish what you want, so in this
case, let's publish name and features. And that's it
on the publisher's side, and so let's go ahead
and do the, or, write the subscriber. So the
only thing you need to do is create a
model with exactly the same name - so, Package.
You know, we're gonna store it in Mongo.
Include the Promiscuous subscriber, so we're, you know, reflection
of the Promiscuous publisher. And then mirror, again, the
publish block with the subscribe block. And let's only
subscribe to the, the name. Just for this example.
And then let's print out the name every time
the name changes. OK.
So that's it. So that's all the code you
have to write. So let's start up the Promiscuous
subscriber and take a look at RabbitMQ.
So RabbitMQ, it's an awesome admin AP- UI, and
we just started the subscriber. There are no queues
here. We should see a queue any second now.
I type bad.
There we go.
So we have a queue in, in Rabbit. We're
gonna kill the worker. Let's go back to Rabbit,
and this is a reason that we, we, we
chose Rabbit is it supports persistent queues. And what
that means is, even if a worker dies, your
queue is still there. It's pers- it's durable. And
any messages that would be consumed or buffered by
Rabbit, and when you start the worker up again
you get those messages.
So you guarantee that those messages are gonna be
delivered.
And that's a property that Rabbit gives you. It's
awesome. So let's go back and let's, let's publish
some of these. So first let's make sure we
save this. Let's reload. Just to get those changes.
And then I want to also illustrate how dependencies
are tracked, so let's do a read, so I'm
just gonna read the last member that we have
in our database, and then let's actually create a
package.
Ok. So there we go. We've created that package
on Crowdtap. Let's, let's pop over back to Rabbit
and take a look at the payload of that
message.
And so here it is. So you can see
a lot of metadata here. Types is, you know,
the class we support inheritance, so you can, you
can publish an entire inheritance tree. You can publish
embedded documents. If you're using Mongoid, that's all supported.
Which, you know, is very powerful. Obviously, the ID,
the attributes that changed, the operation, here's some more
metadata. Importantly in the namespace so it's the name
of the app. Crowdtap. Some more information helps debugging,
like the host that was published from. Current user
if you're in a controller and you're, you're logged
in. Publish that. That's really useful for debugging, and
then, the most interesting part is the dependencies.
So we did a read in, in console, and
here's the associated dependency on that user. So here's
the ID and the version. WE use versioning to
make sure that when this message is consumed on
the subscriber's side, that user exists in that exact
state.
So that's how we manage all the dependencies and
do causal consistencies, and here's the right dependency which
is the, the creation, the object that we just
created.
So that's it. That's the payload. So let's, let's
go ahead and subscribe. And so what we should
see here is you know, hello world being printed
out. Drum roll.
Magic!
OK, so let's, you know, let's show that, let's
show update. OK. Much easier to type when you're
in your room than in front of people. OK.
OK.
All right. Sorry guys.
OK. Well. For some reason this isn't working. But,
we, let me try once more.
OK.That's weird. I'm like typing. OK. Let's, if, if,
if I had typed any change, it would be
immediate propogated over to the publisher's side and trust
me, it works. I can show you after.
So let's, you know, so, so, let's, you know,
what did this to us? So we started, you
know, with these two applications. The main app, which
is Crowdtap, and the targeting app, which is, you
know, Sniper, because it's targeting. And then we pulled
out all of our analytics in fiancé. You now
fiancé because it's tracks engagements, obviously.
And Bobby, you know, because he's the policeman that
makes sure that, you know, when people are posting
stuff, that you know, it's not spam or, and
if it's awesome, that it gets starred. So, you
know, we can iterate on using, you know, AEI
techniques here to, to moderate our, our, our posts.
And then Paparazzi, you know, is out there scraping
Facebook and Twitter and gathering, you know, data, so
that we can report on that for our clients.
And then Casanova, you know, generates CSVs and it's
Casanova because, because it works, because it spreads the
love. I still don't, I still don't get that
name.
But. You know. WE have creative developers. And then
email service, Iris, which is the, the, the nor,
the notification God, and then most interestingly is, at
least in my perspective is Captain Planet. And Captain
Planet came out of a two day hack-a-thon that
we recently had, where a developer was able to,
on his, on his laptop, subscribe to the production
database, production Rabbit, and generate ten reports off of
that data to solve a really big pain point
that we have that our account team has, and
actually do that in a completely isolated way that
didn't impact the production system at all.
And was actually running that on his laptop, just
like the fire hose of production data, on his
laptop, and did that in a couple of days
and worked with it. That is, that's joy, right.
That's back to the joy of the fifteen minute
blog. That's taking a really complex system and making
it, you know, easy to use, and it took,
you know, this, and went back to something ugly
to a lot of little cute fun cuddly apps
that we, that we now have.
And with that, I'm gonna pass it back to
Nico to wrap things up.
N.V.: So to wrap things up, we've seen that
Promiscuous is very useful to do event-driven services. So
you can trigger callbacks realtime. For example, you want
to send emails when some piece of data is
changing. You want to keep your caches warm. Whenever,
or, say, you know, whenever some piece of data
changes, you wanna is, instead of invalidating your caches,
you can actually reconfig the, the right thing.
You could push data asynchronously to your browsers to
improve the user experience. And we want to do
all that also by, and also doing database replication,
but different kind of databases, right. So I want
to be able to replicate my PosGres database to
ElasticSearch and maybe MongoDB and maybe Riot - whatever
it is. And I want this to be decoupled,
and I want this to be, do this at
scale and with good consistency.
So that was gonna allow you to, to really
see the data that you want for the feature
that you need to implement. And so real time
is really, really important these days. And, and if
you can embrace the arch, the, the asynchronous layer
of your application and to really, to really unleash
the power of your data, right. So you can
really innovate now. You can hire some interns for
a summer and get them to do some matching
learning project directly in production without having you to
be worrying about, oh my god, they're gonna drop
my system and they're gonna pollute the code and
all that.
Like, if you don't like their stuff, eventually you
just delete their app and you're done, right. So,
so you can really go back to being agile
and try things really with, without having, oh my
god it's, I need to change so much things.
Like, you, you get back to, to really the,
to the agility of, of, of your team.
So that's, that's about it. So we invite you
to try it out. It's on GitHub, on Promiscuous
dash io slash Promiscuous. So we've been using it
in production for a year now and we're very
happy with it. But it requires a bit of
babysitting for now.
So we're still working on it, and we're trying
to make it really a hands-free experience. So that
you can just plug it in and you have
nothing to worry about and you have all the
different recovery mechanism that kicks in at the right
time and everything.
So if you feel adventurous, you can try to
extract your email logic out of your main application
because that's, like, the obvious, like, you have no,
it's an observer, really.
And, and perhaps build some reporting service, also, for
some, some of your fancy clients. So, if you
try it out, we'll be super happy to help
you with the, to go, with Promiscuous. And we,
you can find us on Twitter and GitHub, kareemk
and nviennot.
Thank you. And now we'll take questions.