JD – Sr. Software Engineer- BigData
The Sr. software engineer will be responsible to design efficient, adaptable, & reusable code. The role requires to tuning components for high performance & scalability using techniques such as caching,code optimization, and efficient memory management. This will involve working with architecture,application development and operations to help achieve a state-of-the-art environment that meets current and future business objectives.
- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
- Monitoring performance and advising any necessary infrastructure changes
- Defining data retention policies
- Gather and process raw data at scale (including writing scripts, write SQL queries, etc.).
- Work closely with our engineering team to integrate your amazing innovations and algorithms into our production systems.
- Process unstructured data into a form suitable for analysis.
- Support business requests for ad hoc analysis as needed.
- We use AWS extensively, so experience with EMR and other web services will help you hit the ground running.
- Should be able to manage and guide team.
- Proficient understanding of distributed computing principles
- Management of Bigdata clusters.
- Ability to solve any on going issues with operating the cluster
- Proficiency with Hadoop v2, MapReduce, HDFS and Big Data querying tools, such as Pig, Hive,and Impala
- Experience with integration of data from multiple data sources.
- Experience with any NoSQL databases, such as HBase, Cassandra, CouchBase, Redis, SSDB,MongoDB
- Experience with any Big Data ML toolkits, such as Mahout, SparkML, or H2O
- Strong knowledge of and experience with statistics; potentially other advanced math as well.
- Programming experience, ideally in Python or Java, but we are open to other experience if you're willing to learn the languages we use.
- Deep knowledge in data mining, machine learning, natural language processing, or information retrieval.
- Experience processing large amounts of structured and unstructured data.
- Enough programming knowledge to clean and scrub noisy datasets.
Qualification: Minimum BE or B Tech or equivalent
Experience: min 7-8 years of work experience