Usually, data pipeline requires complex workflow. Generated data should be sent to various endpoints, and needs to be manufactured by status while moving on.
For example, if there’s a log file stored in S3, the pipeline may need to
Send this to ELK, for monitoring purpose every 10 minute Format to filter useless columns, and send to BigQuery for researching insights, every hour.
Amazon Athena is a service to analyze data stored in S3 with interactive SQL query. You don’t need additional setup, and only need to pay for the queries you’ve run.
It is usually being used by data scientists, or business developer who needs to get insights from big data(probably stored inside S3, or else).
Few days ago, I’ve planned to design some side project, which is based on mobile application. This is the record about how I’ve implement authentication logic through application side to back-end side.
I’ve once wrote about authentication with golang, but in this time I’ve decided to use nodejs.
I’ve had a change to join on hackathon event few weeks ago, and participate as developing backend with infra setup. In my mind hackathon event has more meaning on testing experimental things instead of making stable result, so I’ve decided to try on ‘Cloud Run’.
If you are starting to make some kind of web(or app) service, it is mandatory of thinking about authentication.
Every service has information to show, and it should divide public/private information to be displayed. If you are thinking of financial service, certification and security becomes even more important.