To meet this challenge, we needed an agile platform. It had to be possible to make the changes in an agile, automatic manner, be certain that there would be no production problems.
Thanks to the development of the platform as a code (Cloudformation and Ansible), three identical environments were achieved: Development, integration and production. Minimising potential failures when deploying changes.
Given that the data was the reason for the project, the decision was made to adopt an Event-driven architecture so that all the processes required to exploit the data and make it available were based on events generated by the previous process. For example, on receipt of the data deposited by the original source in S3, the event was generated that enabled the file validator to begin its task.
Just as the data plays a leading role in the project, cost minimisation and processing speed are the major milestones. To achieve these, serverless solutions such as AWS Lambda or AWS EMR were chosen, which are solutions that do not require a 24/7-active node and are only enabled when required (in this case, due to certain events). Support was also given to the development teams to improve the implementation of the developments (parallelised in Elastic MapReduce jobs) or help them adapt to the new platform.