Behind the scenes of eve
The world of software development has continued to change rapidly since its inception. From punch cards, to automatic memory management, to the web. We now exist in an era of services. Decades ago, it was great when you could import the Boost library and use it as a foundational layer for building. While it let you search an array easier, searching an entire dataset still would need an implementation. Today you can just include an Algolia (search) integration, and you do not need to worry about anything except sending and receiving the data. Modern software is built on the tools developed by each generation's predecessor. There is no need to reinvent and build a wheel when you can now just pick one up. However, picking the right wheel can be tricky.
This blog post will deal with the services we use behind the scenes that allows us to build our products rapidly. There are a lot of technologies built into the app itself, see previous blog posts for that (starting here). To build quickly in today's environment, we do not just need a good technological base, but good set of platform integrations and process management services. These giants will help carry us rapidly forward. Some of these giants may be small in some respects, but they are all powerful.
The services will be divided into two primary categories. First those used to make sure our product runs smoothly and efficiently (E.G. Algolia search integration). The second are those that improve our internal velocity (E.G. GitHub repository management).
Search is hard. Very hard. Luckily, there are plenty of tools out there to help with search. From low level Apache Lucene all the way up to Algolia. It all depends on how many pieces of the search puzzle one needs solved. At eve, we wanted all the pieces solved and grouped in one convenient package. Not only does this reduce the likelihood of an implementation bug, but allows us to have a quicker time to market. Fortunately, we do not need a high level of customization on our search, so Algolia is a perfect fit for our current needs.
One of the great features of the eve Control Hub is the real-time updates. There is no need to refresh the page to see new items coming in, they appear automatically along with any edits other users have made. Traditionally real-time is implemented with WebSockets or long polling to the app servers. However, this is not straightforward to set up and can lead to complication when routing connections through a load balancer. Using Pusher allows us to avoid the hassle. After setting up an environment in Pusher, we just need to configure the application to use the correct endpoints for sending and receiving notifications. Our choice to use Laravel makes it even easier, with built in broadcasting support.
Sentry is a great tool for capturing all the errors that happen on our platform. It allows us to see a clear picture of what happened and where in the code it happened. Is a customer encountering a bug? First step is to check Sentry and find an error report that lines up. Often this also give a clear picture of how to reproduce the error, so when testing we can validate the fix. Nothing frustrates a user as much as lingering bugs, so we try to keep eve frustration free using Sentry.
Sentry, as an error platform, unfortunately doesn't capture all the information for future issues. Typically that task is swept aside to logging. Into the logs we dump all the information that might be useful in the future to be able to answer a plethora of questions that engineers, product manager or maybe even marketing want to discover retroactively. The simplest solution is often to just dump the logs to a file on disk, but this presents problems in access management and longevity. There are plenty of good alternative sinks to dump the logs into. For us, we went with Datadog because of the ease of integration with our current infrastructure.
However, at this point we are only skimming the surface of what Datadog can do. Maybe once the platform grows more we will start using all the tools for metrics, infrastructure monitoring and more. But, as is always the case with startups, we will see. 🙂
AWS, Heroku and Netlify
Finally, where does the platform all run? Well, the cloud of course. Physically the servers all currently sit in AWS server farms. However, the actual management of the backend servers is left to Heroku's PaaS (platform as a service). Heroku allows us to not worry about any system maintenance nor any database maintenance. It all happens automatically for us behind the scenes.
For hosting the front end, we do not directly need a server, rather a CDN (content delivery network). We host our static files on Netlify. It is fast and convenient. Similar to Heroku, the primary thing one needs to do to get set up is give it a link to the Git repository. As a side note, an alternative to Heroku and Netlify we have considered is Render as it looks very promising. However, it is still not quite mature enough and does not support a couple of items we need.
Finally, for all the other small infrastructure pieces (DNS, queues, etc.) we use AWS. They have been the largest player in the cloud infrastructure space for a while. While the majority of other large tech companies have good competing offerings, none can match the pricing and expansiveness of AWS.
We need to send several emails throughout our flows, including registration emails and insurance approvals. Mailtrap is a great email capture tool that we use for our staging and local environments. It provides a safe way for the code to send real emails, but at the same time does not risk accidentally sending out emails to real people. When we need to test or verify anything related to the email, we can check the shared inbox for the internal testing servers. In essence integrating with Mailtrap is just a configuration change, so no separate code paths for staging vs production. The simpler the better and less prone to error.
When deciding the direction to develop our products, there are a lot of questions we need to ask and growth metrics we need to track. To allow access to aggregate data views for business development, we use a nice open source tool named Metabase. It allows us to build aggregate views and graphs using a simple interface (or SQL if you need something complex). The visualization tools are great for building dashboards so that we can see our progress with a quick glance.
Cypress is an amazing front end testing tool. No one has quite cracked the secret for front end testing, but Cypress comes close. As we have mentioned previously, we use Cypress for end-to-end testing. The reason it is mentioned again, is it not only provides a package our technology integrates with, but a dashboard to see a clear overview of the tests. This increases our confidence in our sites stability and should some test fail, it makes it easier to dig into the exact point which that occurs.
Last summer we saw an opportunity to help the industry as things opened after lockdown and created an elegant way for businesses to comply with track and trace programs. (You can see evePass for more information.) This was our first mobile app launch, so to make sure it worked across devices we used BrowserStack to test and validate the application. It also helped tremendously during some bug reports to replicate the user experience and quickly identify fixes. As we have wrapped down focused development on evePass, we don't use BrowserStack much anymore. However, keep your eyes peeled for future announcements. We may be needing it again soon 😉.
Internally we use Jira for our process management. Jira is a complex beast. The customization it allows is second to none, it can be morphed into almost any process. Currently at eve we have a fairly basic needs for a ticket management system, but Jira serves that well. Also, as we scale, we will have new needs and there is no doubt that Jira can morph to satisfy those needs.
In the original draft for this blog, GitHub was missed. It has become a staple that we take for granted nowadays, but it is a critical piece of our internal process. We use GitHub for all the typical things, to store our code repositories, review code, and run our continuous integration. It gives us an interface to easily review pull requests, check our build status, and even automatically update our dependencies (via Dependabot). It is almost impossible to find a modern tech startup without GitHub at its core.
And that is it! There will be plenty more services we use or switch to as we grow, but for now we hope this gave a good snapshot of our current system.