Sharing thoughts
2022-01-02 An idea how the Cloud version of my application is going to be.
- WebScraping of training data - Azure Function (Python)
- Store collected data - Azure Cosmos DB
- Data preparation - Databricks and Azure Data Factory
- Training a model and generate inference model - Azure Automated ML as low code as possible( later may be Databricks and Azure ML SDK)
- Getting instances to predict - Azure Function (Python)
- Ad hoc analysis - Azure Synapse Serverless pool
- Visualization - PowerBI
- Versioning and deployment - Azure Devops
2022-01-01 A good day to begin with something new! So I am going to write down here whatever I think is worth remembering. In the free time between two jobs I did what I find more entertaining than watching Netflix or even playing chess. I am busy with converting my MMA classifier to Azure, The first step is to use serverless Azure functions for web-scraping instead of locally running the Python code. I use Visual Code for development, Azure Devops repo voor versioning and Azure pipeline for deployment. After quite a number of attempts, mainly struggling with Python dependencies install at deployment, its finally working. What a joy! Azure Function App scrapes Sherdog.com for UFC events and load the data into a contaner in CosmosDB. Why the heck there? Why not? I must say I find loading JSON into a CosmosDB container without worrying about defining the schema first somewhat liberating. Schema on Read is the way to go. Or may be I am just tired of thinking before I can load data in a database like SQL Server. "Just do it!".
