Logs and metrics in data science and AI

Logs and metrics in data science and AI

ยท

3 min read

Table of contents

๐Ÿ…ธ๐Ÿ…ฝ๐Ÿ†ƒ๐Ÿ†๐Ÿ…พ๐Ÿ…ณ๐Ÿ†„๐Ÿ…ฒ๐Ÿ†ƒ๐Ÿ…ธ๐Ÿ…พ๐Ÿ…ฝ:Data science and machine learning both require data and more accurately a bunch of information. Logs and metrics are two great types of data collecting techniques. These ways don't leave single info without being unnoticed. Both these have different use cases and different way of representation. Over the internet, you can find different logs and metrics collecting as well as representing tools. Some of them will be suggested below in the article. You will also come to know about what google and Microsoft do with this enormous amount of data.

๐Ÿ…ฑ๐Ÿ…พ๐Ÿ…ณ๐Ÿ†ˆ ๐Ÿ…พ๐Ÿ…ต ๐Ÿ†ƒ๐Ÿ…ท๐Ÿ…ด ๐Ÿ…ฑ๐Ÿ…ป๐Ÿ…พ๐Ÿ…ถ:

๐—ช๐—ต๐—ฎ๐˜ ๐—ฎ๐—ฟ๐—ฒ ๐˜๐—ต๐—ฒ ๐—Ÿ๐—ผ๐—ด๐˜€?

Log files are required to store different events that occurred Or occurring. The event can be like a login, visit a website, download of any kind of media, installing an application etc... Windows basically has Data science and machine learning both require data and more accurately a bunch of information. Logs and metrices are two great types of data collecting techniques. These ways don't leave single info without being unnoticed. Both these have different use cases and different way of representation. Over the internet, you can find different logs and metrics collecting as well as representing tools. Some of them will be suggested below in the article. You will also come to know about what google and Microsoft do with this enormous amount.

Windows events logs have different logs like:

๐—”๐—ฝ๐—ฝ๐—น๐—ถ๐—ฐ๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—ฒ๐˜ƒ๐—ฒ๐—ป๐˜ ๐—น๐—ผ๐—ด๐˜€::The application log records events related to different windows system components., like different driver sand buil-in interface elements present in the system.

๐—ฆ๐˜†๐˜€๐˜๐—ฒ๐—บ ๐—ฒ๐˜ƒ๐—ฒ๐—ป๐˜๐˜€ ๐—น๐—ผ๐—ด๐˜€:: The system log records event's related to programs installed on the system.

๐—ฆ๐—ฒ๐—ฐ๐˜‚๐—ฟ๐—ถ๐˜๐˜† ๐—ฒ๐˜ƒ๐—ฒ๐—ป๐˜ ๐—น๐—ผ๐—ด๐˜€:: This records the logs related to logging in attempts and resource access.

๐—ช๐—ต๐—ฎ๐˜ ๐—ฎ๐—ฟ๐—ฒ ๐— ๐—ฎ๐˜๐—ฟ๐—ถ๐—ฐ๐—ฒ๐˜€?

Metrics are basically evolved from the word measurement. Measurement of different hardware and system required to perform any task. For example, when you play a movie on your laptop different requirements occur like a monitor for display, speaker for audio, RAM to run the movie, Graphics for better performance. The measurement or amount of all these requirements measurement or amount of all these requirements is stored in a file which is called metrics. Metrics are collected and stored at a fixed or specific time.

๐—ฆ๐—ฒ๐—ฒ ๐—ณ๐—ถ๐—ฟ๐—ฒ๐˜„๐—ฎ๐—น๐—น ๐—น๐—ผ๐—ด ๐—ณ๐—ถ๐—น๐—ฒ ๐—ถ๐—ป ๐—Ÿ๐—ถ๐—ป๐˜‚๐˜…::

# vi /var/log/firewalld

log command

๐—ฅ๐—ฒ๐—ฝ๐—ฟ๐—ฒ๐˜€๐—ฒ๐—ป๐˜๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—ผ๐—ณ ๐—Ÿ๐—ผ๐—ด๐˜€ ๐—ฎ๐—ป๐—ฑ ๐— ๐—ฎ๐˜๐—ฟ๐—ถ๐—ฐ๐˜€::

All the logs and matrics files have a way of representation or storing the data. This data is stored in a unique fashion so that it could be used further get information.

Each data unit is seoarated by some kind of delimiter like a comma, semicolon, tab, quotes etc... This representation is resolved by using general expressions.

๐—š๐—ผ๐—ผ๐—ด๐—น๐—ฒ ๐—ฎ๐—ป๐—ฑ ๐— ๐—ถ๐—ฐ๐—ฟ๐—ผ๐˜€๐—ผ๐—ณ๐˜ ๐˜‚๐˜€๐—ฒ ๐—ฐ๐—ฎ๐˜€๐—ฒ::

Microsoft best use of logs is to troubleshoot the computer. These logs help Microsoft to diagnose the problem and resolve it. On the other hand, Google uses history to make things like advertisement and content more personalize and useful. Different events are maintained for different tasks to be performed. Like if you face any problem with the system application, then the log system generated files will be checked.

Both Google and Microsoft have different tools to monitor logs and metrics over the cloud platform GCP and AZURE.

๐Ÿ…ฒ๐Ÿ…พ๐Ÿ…ฝ๐Ÿ…ฒ๐Ÿ…ป๐Ÿ†„๐Ÿ†‚๐Ÿ…ธ๐Ÿ…พ๐Ÿ…ฝ:

A log is a computer generated file that captures activity within the operating system. They improves the accuracy of the model. Metrics are the vital signs that tell you how your devics and applications are performing. Therefore, both of these are called as data collecting techniques.

syednawaz.hashnode.dev

ย