Job Overview
| Particulars | Details |
|---|---|
| Position | Senior Data Engineer – Databricks, PySpark & Lakehouse |
| Company | IDC Technologies |
| Location | India |
| Job Type | Remote |
| Experience | 6 - 7 years |
| Category / Department | IT |
| Application Mode |
🏢 About IDC Technologies
About IDC Technologies
IDC Technologies is a global IT consulting and staffing organization focused on technology solutions, managed IT services, staffing, and consulting.
The company describes itself as a solution-oriented organization that emphasizes meaningful communication, strong relationships, transparency, and delivering technology and staffing solutions aligned with client requirements.
IDC Technologies is headquartered in Silicon Valley, USA, with offices and operations across multiple regions globally.
Services
IDC Technologies provides services across areas including:
IT Consulting
IT Staffing
Managed IT Services
Technology Consulting
Workforce Solutions
Recruitment and Talent Solutions
Engineering and Technology Services
The company works with customers across different industries and focuses on delivering solutions with quality and rapid turnaround.
Company Mission:
IDC Technologies' mission is to become a strong, service-first, trusted, and globally recognized staffing and consulting strategic partner while helping redefine the dynamics of the industry.
Company Vision:
The company's vision is to build centers of excellence that provide high-quality experiences for clients and partners while maintaining strong standards of:
Integrity
Quality
Mutual Respect
Professionalism
Service Excellence
Company Culture:
IDC Technologies emphasizes:
Human connection
Relationship building
Transparency
Integrity
Teamwork
Flexibility
Customer focus
Talent development
Long-term partnerships
IDC Technologies
Near Gattani Petrol Pump
Arjun Ganj, Sultanpur Road
Lucknow – 226002
Uttar Pradesh, India
💡 Editor's Note / Preparation Tip
Resume Tips:
Senior Data Engineers should prominently highlight:
Databricks
PySpark
Apache Spark
Spark Structured Streaming
Auto Loader
Delta Live Tables
Delta Lake
Unity Catalog
Lakehouse Architecture
ETL/ELT
Batch processing
Streaming pipelines
Data governance
Data quality
AI/ML data pipelines
GenAI projects
Include measurable achievements such as:
Data pipeline performance improvements
Processing-volume improvements
Reduction in pipeline execution time
Cost optimization
Streaming latency reduction
Data quality improvements
Migration to Databricks
Lakehouse implementation
Large-scale data processing
Interview Preparation:
Prepare for questions covering:
PySpark
Spark architecture
Spark transformations vs actions
RDD vs DataFrame vs Dataset
Spark partitioning
Shuffle operations
Broadcast joins
Caching
Spark optimization
Adaptive Query Execution
Structured Streaming
Checkpointing
Watermarking
Output modes
Databricks architecture
Auto Loader
Delta Lake
Delta Live Tables
Unity Catalog
Medallion Architecture
Lakehouse Architecture
Data governance
Data quality
ETL/ELT
Batch vs streaming
Real-time pipelines
Performance optimization
Cloud data platforms
AI/ML data pipelines
GenAI data architecture
Professional Tips:
Candidates should be ready to explain:
Their largest Databricks project.
How they designed a Lakehouse architecture.
How they implemented streaming pipelines.
How Auto Loader was used for ingestion.
How they implemented DLT.
How Delta Lake improved reliability.
How Unity Catalog was used for governance.
How they optimized Spark workloads.
How they handled pipeline failures.
How they designed scalable data platforms.
Their experience supporting AI/GenAI workloads.
📝 Job Description
Position Overview
IDC Technologies is looking for an experienced Senior Data Engineer to join its data engineering team in a fully remote capacity.
The successful candidate will work with modern cloud data engineering technologies and contribute to building scalable data platforms based on Databricks and Lakehouse Architecture.
The role requires strong hands-on expertise in distributed data processing, streaming data pipelines, Delta Lake, data governance, and modern Databricks technologies.
Core Technology Stack
The primary technologies include:
Databricks
PySpark
Apache Spark
Spark Structured Streaming
Databricks Auto Loader
Delta Live Tables (DLT)
Delta Lake
Unity Catalog
Lakehouse Architecture
AI/GenAI Solutions
Experience Required
7+ years of professional experience in data engineering or related technology roles.
Candidates should have substantial hands-on experience building and maintaining data platforms and pipelines using Databricks and Apache Spark technologies.
Key Technical Areas
The ideal candidate should understand:
Distributed data processing
Batch data processing
Real-time data processing
Streaming architectures
Data pipeline development
Lakehouse architecture
Delta-based data architectures
Data governance
Data quality
Scalable data engineering
AI/GenAI data solutions
Work Environment
India – Remote
The position allows the selected candidate to work remotely from India.
Selection Process
Only shortlisted candidates will be contacted.
How to Apply:
Interested candidates can send their updated resume to:
mounika.t@idctechnologies.com
🎯 Key Responsibilities
- ✓ Design and develop scalable data engineering solutions.
- ✓ Build and maintain data pipelines using Databricks and PySpark.
- ✓ Develop batch and streaming data processing workflows.
- ✓ Implement Spark Structured Streaming solutions.
- ✓ Build ingestion pipelines using Databricks Auto Loader.
- ✓ Develop and manage Delta Live Tables (DLT) pipelines.
- ✓ Implement Delta Lake-based data architectures.
- ✓ Work with Unity Catalog for data governance and access management.
- ✓ Design and implement Lakehouse Architecture.
- ✓ Optimize Spark jobs and data processing workloads.
- ✓ Handle large-scale datasets efficiently.
- ✓ Develop reliable and reusable data transformation frameworks.
- ✓ Implement data quality and validation processes.
- ✓ Troubleshoot pipeline failures and performance issues.
- ✓ Support real-time and near-real-time data processing.
- ✓ Collaborate with data scientists, ML engineers, software engineers, and business teams.
- ✓ Contribute to AI and Generative AI-related data engineering solutions.
- ✓ Follow data engineering best practices and architectural standards.
- ✓ Monitor data pipelines and improve reliability.
- ✓ Ensure data solutions are scalable, secure, and maintainable.
💡 Required Qualifications & Skills
🚀 Career Guidance & Interview Insights
To help you succeed, we've compiled original preparation guides, resume keywords, and growth analytics for this category of role.
Joining as a Senior Data Engineer – Databricks, PySpark & Lakehouse offers a fantastic opportunity to work at the forefront of technological innovation. You will be exposed to modern software development lifecycles, collaborate with cross-functional product teams, and design scalable architectures that directly impact end-users. The continuous learning curve in tech ensures that your programming, debugging, and system engineering skills remain highly marketable and sharp.
The career roadmap in technology is highly rewarding. Typical progression starts as a Junior Software Engineer, moving to Associate Developer and then Senior Software Engineer (3-6 years). From there, professionals choose between an individual contributor track—becoming a Tech Lead, Staff Engineer, and Principal Architect—or a management track, advancing to Engineering Manager, Director of Engineering, and Chief Technology Officer (CTO).
Expected Technical & Behavioral Questions:
- Q1: How do you optimize database query performance?
Answer Tip: Talk about indexing strategies, query execution plan analysis, avoiding N+1 queries, and implementing caching layers (like Redis). - Q2: Explain the differences between Microservices and Monolithic architecture.
Answer Tip: Highlight scaling advantages, loose coupling, independent deployments, network latency considerations, and data consistency challenges in microservices. - Q3: How do you handle merge conflicts in Git?
Answer Tip: Explain checking out the local branch, pulling the latest main branch, running git merge, opening the conflicted files to resolve blocks manually, and committing the resolved merge.
To pass automated ATS (Applicant Tracking System) screening and catch the recruiter's eye, tailor your resume with the following tips:
- Keywords to include: Software Engineering, SDLC, OOP, CI/CD Pipelines, SQL databases, Agile, Unit Testing, Code Review, Cloud Architectures (AWS/Azure).
- Format: Use a clean, single-column resume format. Avoid graphics, text boxes, or tables which can scramble ATS parser outputs.
- Quantify Results: Instead of writing 'wrote code', write 'Developed a caching system that reduced server response times by 35%.'
Estimated compensation for this role type in India is ₹12,00,000 - ₹25,00,000+ per annum (Senior/Lead level). The actual salary package offered depends on factors such as company size, work mode (remote or on-site), individual technical proficiency, and negotiations during final HR rounds.
Highly recommended certifications include: AWS Certified Solutions Architect, Microsoft Certified: Azure Fundamentals, Certified Kubernetes Administrator (CKA), Oracle Certified Professional Java SE, or tech-stack specific achievements (e.g. Meta Front-End Developer certification).
Frequently Asked Questions
1. What position is IDC Technologies hiring for?
IDC Technologies is hiring for a Senior Data Engineer position.
2. How much experience is required?
Candidates should have 7+ years of experience.
3. Is this a remote opportunity?
Yes. The position is advertised as India – Remote.
4. What are the main technologies required?
The primary technologies are Databricks, PySpark, Spark Structured Streaming, Auto Loader, DLT, Delta Lake, Unity Catalog, and Lakehouse Architecture.
5. Is PySpark experience required?
Yes. Strong hands-on PySpark experience is one of the key requirements.
6. Is Spark Structured Streaming required?
Yes. Experience with Spark Structured Streaming is specifically mentioned.
7. What is the role's experience requirement?
The role requires 7+ years of experience.
8. Is Delta Lake experience required?
Yes. Candidates should have experience with Delta Lake.
9. Is AI/GenAI experience relevant to this role?
Yes. Experience with AI/GenAI solutions is included among the required areas.
10. How can I apply?
Send your updated resume to mounika.t@idctechnologies.com. Only shortlisted candidates will be contacted.
Share This Job
🛡️ Safety Disclaimer: JobPortalConnect curates official listings for informational purposes. Genuine recruiters will never ask you for processing fees, training payments, or security deposits. If you spot suspicious activity, copyright issues, or wish to request the removal of this listing, please report it to us immediately at contact.jobportalconnect@gmail.com.