Automated web scraping tutorial using jsoup, JPA, EclipseLink and ADF Essentials 12.1.3

Q: How does a programmer surf the web? A: With a scraper.

There are several sites I visit periodically to track a specific set of data relevant to my interest. This rinse and repeat process quickly becomes boring and prone to error. Now I started thinking as a programmer…if only a piece of software can do this for me, I can spend the free time on drinking coffee and surfing other sites! Apparently this is pretty much what a web crawler does and more specifically “scraping” is the term used to describe targeted searches. Below is a two part video on a couple of scraping examples I’ve built and a basic tutorial on how to build your own.

Part 1: Introduction

Part 2: Implementation

To run the main class in a Linux env:
java -cp “jsoup-cronjob-pubmed.jar:lib/*” com.adfhomebrew.jsoup.cronjob.pubmed.PubMedSearchClient

Jars needed:
TopLink/EclipseLink: JDEV_INSTALL\oracle_common\modules\oracle.toplink_12.1.3\eclipselink.jar

JPA: JDEV_INSTALL\oracle_common\modules\javax.persistence_2.0.jar

mysql jdbc driver: http://dev.mysql.com/downloads/connector/j/5.0.html

jsoup: http://jsoup.org/download

The live examples and links to tutorial code can be found at http://www.adfhomebrew.com

Posted in Uncategorized | 1 Comment

ADF Directed Acyclic Graphs

One of the application I created a prototype for at work collects surveys which patients fill out. An example of a simple survey question with “branching” requirement can be related to gender. Based on that answer, only related sub questions should be posed. In order to model this dynamically(vs hard coding a survey), I had to revisit a topic from discrete math, specifically the concept of a directed graph. By implementing the underlying logic using a graph, a generic approach can be used to guide a user to fill out a survey for only relevant questions. There are many other use cases where graphs drive the underlying application logic (GPS, Clinical Informatics – SNOMED CT, Bayesian Networks, Supply Chain Delivery, etc..) and it’s up to you as the engineer to decide on the appropriate application.

Below is a short demo of the graph examples I have created:

Live demo
http://www.adfhomebrew.com/home/faces/graph2

Code via GITHUB
http://github.com/nycjdev/adfhomebrew

Posted in Uncategorized | 3 Comments

Solution: ADF Faces 11.1.2.4, How to programmatically build a dvt:pieGraph with List of elements.

While looking for a programmatic approach to generating a dvt:pieGraph, I noticed the collection data is slightly different compared to rendering a dvt:barGraph. After a quick search on the OTN Forum returned a thread but it looks like it was never answered entirely.

See the example solution below:

Within the backing bean:

public List<Object[]> getListObject() {
if (listObject == null) {
listObject = new ArrayList<Object[]>();
Object[] obj1 = { "Example_Bar_1", "Series_1", 10 };
Object[] obj2 = { "Example_Bar_1", "Series_2", 15 };
Object[] obj3 = { "Example_Bar_1", "Series_3", 75 };

listObject.add(obj1);
listObject.add(obj2);
listObject.add(obj3);

}
return listObject;
}

JSF code:

<dvt:pieGraph id="graph2" subType="PIE" tabularData="#{YOURBEAN.listObject}">
<dvt:graphPieFrame/>
<dvt:sliceLabel/>
<dvt:pieLabel rendered="true"/>
<dvt:legendArea automaticPlacement="AP_NEVER"/>
</dvt:pieGraph>

Runtime render:
Image

Posted in Uncategorized | Leave a comment

Tutorial: Declaratively render UTF menu model using various ADF Faces components.

Screen shots
menus2
menus

Trailer video of end product

The idea for this tutorial started with a UI requirement for our system to provide navigation flexibility at runtime to various modules. Also our clinical business analysts wanted a “Site map” option to offer a global view for all our UTF pages. The solution is to simply use the XML menu model generated by JDeveloper to provide runtime rendering of the menu and the site map without repeating the underlying model code.

Below is the tutorial video:

Also check out this nice tutorial on building dynamic menus using Navigation Panes

Beyond this demo, you should be able to incorporate this component into a UI template or even a declarative component to fully leverage reuse within your application. Note that after generating and modifying the xml menu model, JDev does not know how to sync additional changes to your UTFs and you will need to update the menu model separately. Happy coding!

Update 4/10/2014
There were some questions regarding the PanelAccordion implementation and how to keep the tabs disclosed after a link is clicked. A simple way to do this using EL is to define an setActionListener component to the commandlink and set the ID of the parent node. Then perform a comparison within the parent node’s “disclosed” property. See the below code for reference.

<af:panelAccordion id="pa1">
<af:forEach items="#{root_menu.wrappedData.wrappedData}" var="root">
<af:showDetailItem text="#{root.label}" id="sdi1"
disclosed="#{root.id == processScope.currentTabId}">
<af:forEach items="#{root.children}" var="children">
<af:commandLink text="#{children.focusViewId}" id="cl1" action="#{children.doAction}"
partialSubmit="true" immediate="true">
<af:setActionListener from="#{root.id}" to="#{processScope.currentTabId}"/>
</af:commandLink>
</af:forEach>
</af:showDetailItem>
</af:forEach>
</af:panelAccordion>

Posted in Uncategorized | 7 Comments

Start to finish, ADF Essentials deployment on Amazon EC2

Developers! Developers! Developers! … gives you an idea about how excited I am to finally have finished my latest tutorial on how to deploy an ADF application and make it publicly available. One other incentive here is it’s free cloud hosting for a year. I am certainly not here to sell you anything and there are instability issues with the service you can read about on the web but in my opinion it’s a great way to get a feel for setting up your ADF infrastructure from scratch. Hopefully this tutorial can serve as a reference for you to begin your own quest to making your ADF app available to everyone!

The key steps for setting up Glassfish with ADF Essentials has been well documented by Shay Shmeltzer. Check out his informative blog post and his youtube video on the procedure. Below are links to the video series and I apologize in advance for the poor audio quality, my pc microphone is terrible and I’ve put in an order for a new condenser mic for future recording.

Part 1 – Setting up Linux AMI instance
Part 2 – Local software install and ssh/sftp setup.
Part 3 – Remote software install, db & datasource config.
Part 4 – Jdeveloper connection setup, build simple ADF app.
Part 5 – Deployment

Also I have listed below some long commands and related information I used for the setup.

For Cygwin/X packages:
xorg-server
xinit
inetutils
openssh

After installing JDK, added JAVA_HOME to .bashrc file:
export JAVA_HOME=”/usr/lib/jvm/jdk1.7.0_15″
export PATH=”$JAVA_HOME/bin:$PATH”

Mysql account creation:
CREATE USER ‘john’@’localhost’ IDENTIFIED BY ‘mypassword';
GRANT ALL PRIVILEGES ON *.* TO ‘john’@’localhost’ WITH GRANT OPTION;
CREATE USER ‘john’@’%’ IDENTIFIED BY ‘mypassword';
GRANT ALL PRIVILEGES ON *.* TO ‘john’@’%’ WITH GRANT OPTION;

Glassfish connection pool creation command via asadmin:
create-jdbc-connection-pool –datasourceclassname=com.mysql.jdbc.jdbc2.optional.MysqlDataSource –restype=javax.sql.ConnectionPoolDataSource –property=”user=john:password=mypassword:url=jdbc\\:mysql\\://localhost\\:3306/databasename”

That’s it! Let me know if you have any issues or found other ways to simplify any of the given steps, hit me up on twitter @wesleyfang. You can also visit my live instance at http://www.adfhomebrew.com Thanks for looking!

Posted in Uncategorized | 5 Comments

Oracle ADF continuous integration using Hudson and automated Weblogic deployment.

tl;dr

Use the following ant task definition for deploying to weblogic.

<target name="weblogic-undeploy">
    <java classname="weblogic.Deployer"
          args="-adminurl t3://WEBLOGICSERVERURL:7001 -user USERNAME -password PASSWORD -name APPLICATIONNAME -undeploy"
          classpath="${install.dir}/wlserver_10.3/server/lib/weblogic.jar">       
    </java>
  </target>
 
  <target name="weblogic-deploy" depends="weblogic-undeploy">
    <java classname="weblogic.Deployer"
          args="-adminurl t3:// WEBLOGICSERVERURL:7001 -user USERNAME -password PASSWORD  -deploy C:\PATHTOEARFILE\scotttiger.ear -name APPLICATIONNAME -upload"
          classpath="${install.dir}/wlserver_10.3/server/lib/weblogic.jar">       
    </java>
  </target>

It’s been hard to find time between our full throttle ADF 11g development and updating content on my blog but finally after getting back from OOW13 I was inspired to spend some more time on putting together this latest bit about automated weblogic deployment. After following a pair of great tutorials by John Stegeman and Chris Muir I was able to successfully setup our own Hudson build server. Please see the below links I used for reference:

http://www.oracle.com/technetwork/articles/adf/part7-085284.html
http://www.youtube.com/watch?v=IP-etlVfU0Y

The video tutorial series is broken up to four parts and it is intended to visually guide you from beginning to end procedures for setting up your own Hudson build server. Also it ties together the above two resources and includes a couple of unmentioned settings within the ant build script. The only components that are not discussed is setting up a SVN repository & standalone weblogic server instance.

Here is the basic topology & flow of our ADF CI setup:

almcrop

  1. Developer commits their code into svn server.
  2. Hudson server starts a job on a set time interval to poll svn for changes
  3. Upon picking up changes, a build is kicked followed by junit/selenium tests (I will try to put together a post on this in the future).
  4. A successful build will produce an ear file and finally it is uploaded to the standalone weblogic QA server.

Good podcast on what hudson can do for you (added 10/17/2013)

Components I am using
—  JDeveloper 11.1.2.4
—  Weblogic 10.3.5
—  JDK 6u24 (bundled with JDev)
—  Ant 1.7 (bundled with JDev)
—  SVN 1.7
—  Hudson 3.0
—  Glassfish 3.1

Below is a short 4 part video series I made showing the steps described above.

Part 1  Glassfish & Hudson install & configuration.
Part 2  Create and version a simple ADF application.
Part 3 Generate and modify ant build script to create ear file & undeploy/deploy tasks for weblogic.
Part 4 Define a Hudson job for automatic deployment.

I realized that dubbing voice over after the video was created is actually sort of pesky as I need to sync with the recorded actions so I decided to use boring captions but in the future I will include audio. Thanks for reading and feel free to post any questions or suggestions.

Posted in Uncategorized | Leave a comment

Solution: How to setup standalone Weblogic 11g (10.3.6) with ADF Runtime 11g (11.1.2.4)

It seems that I’m always behind the latest ADF Technology, just as I was wrapping up my testing of various ways to setting up a standalone WLS/ADF 11g install, Oracle releases wls/jdev 12c! Thus, if you are looking to get started with setting up weblogic/adf 12c, I highly suggest you begin your quest with the following resources:

But if you are here because you are trying to specifically configure the latest WLS 11g with ADF 11.1.2.4, then hopefully the following 3 part video can be used as reference for the install. Also, if you are trying to install ADF 11.1.2.4 on WLS 10.3.5 (the quickest way), please use this blog post.

Part 1

Files required:

Steps:

  1. Run wls1036_win32.exe
  2. Setup a new domain
  3. Install ADF Runtime 11.1.1.6
  4. Extend domain (from step 2) with EM & JRF

Part 2

Files:

  • p16546129_111160_Generic.zip(metalink)
  • p16546157_111160_Generic.zip(metalink)

Steps:

  1. Install patch 16546129
  2. Install patch 16546157
  3. Upgrade domain to ADF Runtime 11.1.2.4 via wlst

Part 3

Files:

Steps:

  1. Start Weblogic
  2. Start JDeveloper
  3. Define application server connection
  4. Obtain/deploy ADF Runtime Version tester (by Jan Vervecken)
  5. Test!

My OTN Forum question regarding overall approach

 

Posted in Uncategorized | 2 Comments