Data Engineering & Analytics - Info Services

Data Engineering & Analytics

A collection of 10 posts

Datahub Hive Kerberos Authentication

Data Engineering & Analytics

Datahub Hive Kerberos Authentication

Did you know that nearly 70% of enterprises face challenges when managing secure data ingestion from protected environments? With the increasing reliance on tools like DataHub for data observability, the need for secure, seamless metadata ingestion has never been greater. Particularly when dealing with Kerberos-secured Hive clusters, organizations often struggle

Handling Imbalanced Datasets in Anomaly Detection: Best Practices

Data Engineering & Analytics

Handling Imbalanced Datasets in Anomaly Detection: Best Practices

Anomaly detection has become an important part of many different industries today, like finding the trickery in finance and system failure in IT infrastructures. The IDC study forecasts that “The Global Datasphere will expand from 33 zettabytes in 2018 to 175 zettabytes by 2025,” whereas Statista anticipates 180 zettabytes and

A Timely Solution for Time Series Data Management and Analysis with AWS Timestream

Data Engineering & Analytics

A Timely Solution for Time Series Data Management and Analysis with AWS Timestream

Time series data is a critical component of many modern applications, including IoT, financial services, media & entertainment, and more. Managing and analyzing time series data can be challenging due to the high volume and velocity of data, as well as the need to perform complex queries and calculations over

How EMR Serverless and Delta Lake are Transforming Data Management in the Cloud

Data Engineering & Analytics

How EMR Serverless and Delta Lake are Transforming Data Management in the Cloud

EMR (Elastic MapReduce) is a cloud-based service provided by Amazon Web Services (AWS) that allows users to process large amounts of data using distributed computing. It is designed to scale up and down seamlessly, making it a popular choice for big data processing and analytics. One of the newer features

Predict Industrial Equipment Health with AWS SageMaker Algorithms

Data Engineering & Analytics

Predict Industrial Equipment Health with AWS SageMaker Algorithms

Predicting the health of industrial equipment using vibration data from sensors is a common application of machine learning in the industrial sector. By analyzing the vibrations emitted by the equipment, it is possible to identify patterns and anomalies that can indicate the presence of potential problems. One approach to predicting

Build custom models using Amazon Rekognition

Data Engineering & Analytics

Build custom models using Amazon Rekognition

Amazon Rekognition is a machine learning service for image and video analysis. It provides a wide range of features to suit many image and video analysis use cases. Features * Content Moderation: This can be used to detect inappropriate, unwanted or offensive content. * Face compare and search * Face detection and analysis:

Reference Architecture - Serverless Interactive Analytics with Amazon Athena and AWS Glue

Data Engineering & Analytics

Reference Architecture - Serverless Interactive Analytics with Amazon Athena and AWS Glue

Context Serverless applications gained a huge adaption over last few years with AWS offering services like AWS APIGateway, AppSync, Lamda, Amplify etc. We have seen earlier how to develop serverless web applications using Amplify in our previous blog “Developing Serverless applications with AWS Amplify”. In this blog, we discuss how