Thursday, March 5, 2020

Pipe in Spark

Pipe operator in Spark, allows developer to process RDD data using external applications. Sometimes in data analysis, we need to use an external library which may not be written using Java/Scala. Ex: Fortran math libraries. In that case, spark’s pipe operator allows us to send the RDD data to the external application.
In this post, first we are going to look at how we can use pipe operator. Once we understand the usage, then we will see how we can implement pipe operation in normal scala programs. This implementation is taken from spark implementation.

Pipe in Spark


The following steps shows how to use pipe operator. To start with, we will create an RDD from inmemory list. Step 1 : Create a RDD
val data = List("hi","hello","how","are","you")
 val dataRDD = sc.makeRDD(data) //sc is SparkContext  
Step 2 : Create a shell script
Once we have RDD, then we will pipe it to a shell script. Let’s create a file called echo.sh, then put the following content.
#!/bin/sh
echo "Running shell script"
while read LINE; do
   echo ${LINE}    
done

This is a simple shell script which reads the input from stdin and output that to stdout. You can do any other shell operation in this shell script.
Step 3 : Pipe rdd data to shell script
One we have the shell script, we can pipe the RDD through this script. Make sure that you change the scriptPath variable to match path of your file.
val scriptPath = "/home/hadoop/echo.sh"
val pipeRDD = dataRDD.pipe(scriptPath)
pipeRDD.collect() 

Now you should able to see, the line printed on console with echo messages from shell script. In place of shell script, you can use any other executable.

Wednesday, March 4, 2020

Proper use of Spark Shell While Writing the Spark Code


Problem Satement:
I am writing a spark probelm , which I need to read a csv file and convert that file data to an RDD and apply groupBy transformation on that .
While doing that I have written the following code.

scala> case class Customer(name : String, age :Int, gender:String,zip:String)
scala> var customers = customerData.map{line =>{ val a:Array[String] = line.split(",") Customer(a(0),a(1).toInt,a(2),a(3)) }}

When i execute the above statement, am getting the following error.


<console>:25: error: value Customer is not a member of Array[String]
var customers = customerData.map{line =>{ val a:Array[String] = line.split(",") Customer(a(0),a(1).toInt,a(2),a(3)) }}

I did not understand the error , how it is coming even though I have written the correct code ..
after analyzing the code and the source of this code . I came to know that I have written all the code in single line...
So that compiler is not able to understand the Customer Case Class.

Solution:
I have added semicolon(;) after splitting the line.
Then code is working fine.

scala> var customers = customerData.map{line =>{ val a = line.split(","); Customer(a(0),a(1).toInt,a(2),a(3)) }}
customers: org.apache.spark.rdd.RDD[Customer] = MapPartitionsRDD[19] at map at <console>:27


Tuesday, March 3, 2020

To configure EhCache to Spring Boot Application

To configure EhCache to Spring Boot Application

First we have to add the following maven dependency to Application (Note versions will be taken care by spring boot starter (which we have added initial setup))
		
                
			org.springframework.boot
			spring-boot-starter-cache
		

		
			javax.cache
			cache-api
		

		
			org.ehcache
			ehcache
		

Add the following class to application as setup
	package com.nagarajuconfig;

import java.util.concurrent.TimeUnit;

import javax.cache.CacheManager;
import javax.cache.Caching;
import javax.cache.configuration.FactoryBuilder;
import javax.cache.configuration.MutableCacheEntryListenerConfiguration;
import javax.cache.configuration.MutableConfiguration;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import javax.cache.spi.CachingProvider;

import org.springframework.stereotype.Component;

@Component
public class CacheSetup {

	public CacheManager cacheManager() {
		CachingProvider provider = Caching.getCachingProvider();
		CacheManager cacheManager = provider.getCacheManager();

		cacheManager.createCache("cacheName", getCacheConfiguration());

		return cacheManager;
	}

	private MutableConfiguration getCacheConfiguration() {
		return new MutableConfiguration().setTypes(Object.class, Object.class).setStoreByValue(false)
				.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new Duration(TimeUnit.MINUTES, 1)))
				.setStatisticsEnabled(true);
	}

}

Add the following to service layer or dao layer ..I have added in the service Layer.
	@Override
	/* @Cacheable(cacheNames = "controllerParamData",key="#controllerId") */
	@CacheResult(cacheName = "controllerParamData")
	public List getParamData(Long controllerId, String type, String sort, Integer count,
			Long startTime, Long endTime, List paramKey)
Now enable the Cache
	package com.nagaraju.config;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.jcache.JCacheCacheManager;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 
 */
@Configuration
@EnableCaching
public class CacheConfiguration {

	@Autowired
	private CacheSetup cacheSetup;
	
	@Bean
	JCacheCacheManager jCacheCacheManager() {
		return new JCacheCacheManager(cacheSetup.cacheManager());
	}

}

Run application, then will setup the cache automatically.

Wednesday, February 26, 2020

Invalidate Session on Spring @ControllerAdvice

Problem:

When some exception comes in spring web application, some times we have to redirect the page some exception jsp/html page .
we have to fix this problem in ControllerAdvice Class.

Solution:

        
    @ExceptionHandler(ResourceAccessException.class)
    public ModelAndView handleResourceAccessException(ResourceAccessException ex,HttpServletRequest request) {
      LOG.error(START);
      HttpSession session= request.getSession(false);
        SecurityContextHolder.clearContext();
         if(session != null) {
             session.invalidate();
         }
        ModelAndView model = new ModelAndView();
        LOG.error(MESSAGE,ex.getMessage());
        model.setViewName("generic_page");
        LOG.error(END);
        return model;
    }
  • we get the session Object from HttpServletRequest object and checks the null condition and after that we have invalidated the session.
  • Important point

    is that we have write this statment : SecurityContextHolder.clearContext(); otherwise Spring Security will not identify our session invalidation.

Bootstrap Scroll able Menu and selection of data value and text

        
<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
<script src="https://code.jquery.com/jquery-1.11.1.min.js"></script>
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.0/css/bootstrap.min.css" rel="stylesheet" id="bootstrap-css">
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.1.0/js/bootstrap.min.js"></script>


<style>
  .scrollable-menu {
      height: auto;
      max-height: 200px;
      overflow-x: hidden;
  }
</style
</head>
<body>
<div class="container">
    <div class="row">
        <div class="col-lg-12">
            <h2 class="bold">Scrollable Menu</h2>
            <div class="btn-group">
                <div class="dropdown">
  <button  style="max-width: 300px;width: 200px;"class="btn btn-default dropdown-toggle" type="button" id="dropdownMenu1" data-toggle="dropdown" aria-haspopup="true" aria-expanded="true">
    Dropdown
    <span class="caret"></span>
  </button>
  <ul class="dropdown-menu scrollable-menu" aria-labelledby="dropdownMenu1">
    <li><a href="#" data-value="Nagaraju">Action</a></li>
    <li><a href="#" data-value="another action">Another action</a></li>
    <li><a href="#" data-value="something else here">Something else here</a></li>
    <li><a href="#" data-value="separated link">Separated link</a></li>
<li><a href="#" data-value="action">Action</a></li>
    <li><a href="#" data-value="another action">Another action</a></li>
    <li><a href="#" data-value="something else here">Something else here</a></li>
    <li><a href="#" data-value="separated link">Separated link</a></li>
<li><a href="#" data-value="action">Action</a></li>
    <li><a href="#" data-value="another action">Another action</a></li>
    <li><a href="#" data-value="something else here">Something else here</a></li>
    <li><a href="#" data-value="separated link">Separated link</a></li>
  </ul>
</div>
            </div>
         

        </div>
    </div>
</div>
</body>
<script>
  $(function() {
  $(".dropdown-menu li a").click(function(){
    $(this).parents(".dropdown").find('.btn').html($(this).text() + ' <span class="caret"></span>');
    console.log($(this).attr('data-value'));
    $(this).parents(".dropdown").find('.btn').val($(this).data('value'));
  });
  })
</script>
</html>

User creation on MySQL and giving Privileges


mysql> CREATE USER 'good'@'localhost' IDENTIFIED BY 'man';
Query OK, 0 rows affected (0.16 sec)

mysql> GRANT ALL ON iot.* TO 'good'@'localhost';
Query OK, 0 rows affected (0.18 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.12 sec)
-------------------------------------------------------------------------------------------------------------------------

Here localhost means that  Data Pushing Application and MySQL DB Server both are on the same System.
--------------------------------------------------------------------------------------------------------------------------
If Data Pushing Application  and MySQL DB Server are on different Systems.

Then we give  the permission to that  Data Pushing Application like ..
CREATE USER 'good'@’2.16.1.200’ IDENTIFIED BY 'man';
GRANT ALL ON iot.* TO 'good'@'2.16.1.200';   (Here 2.16.1.200 is the Data Pushing Application IP Address.)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
If we want to access any host(IP Address of any System) then
 CREATE USER 'good'@’%’ IDENTIFIED BY 'man';
GRANT ALL ON iot.* TO 'good'@'%';   (Here % means that Any host can access our MySQL Server DB)
-------------------------------------------------------------------------------------------------------------------------

Tuesday, February 25, 2020

Learning any Technology


The following are the main things to learn any technology.


  1. Go to its official Web Site and learn the Getting Started and then read the complete documentation.
  2. Read any best book on that technology.
  3. Teach to any body about this technology which you have learned.
  4. Practice that technology.
  5. Revise the technology at some intervals.



Recent Post

Databricks Delta table merge Example

here's some sample code that demonstrates a merge operation on a Delta table using PySpark:   from pyspark.sql import SparkSession # cre...