Python - Creating Module Packages

Intent

Beyond importing someone else's packages, sometimes it helps to make your own. This will assist in the understanding of how to do so.

I. It helps to have the pep8 compliance package to test.

py -3 -m pip install pytest
py -3 -m pip install pytest-pep8

Next step, make sure your module is pip complaint

1. cd to your module and run…
2. py.test --pep8 vsearch.py

Common Errors Found

1. Not having two blank lines between each function
2. Not having a white space following a colon
3. Using tabs to indent. It prefers 4     blank   manual    lines

Note the ^ will mark where in the code it believes things are out of compliance.

II. Next a setup.py and a README.txt file are required

 

Creating setup.py

 

Textbook example:

from setuptools import setup

setup(name='vsearch', version='1.01', description='The Head First Python Search Tools', author='HF Python 2e', author_email='hfpy2e@gmail.com', url='headfirstlabs.com', py_modules=['vsearch'],)

Creating README.txt

Just put in whatever you want. So long as the README exists.

III. Run setup.py as the setup distribution

In the command line run as follows: 

py -3 setup.py sdist

IV. Install your new module from within it's distribution

Example:

py -3 -m pip install [moduleName]-1.0.zip

V. Now you can run your module. Don't forget to run it as something, or you'll have to call it by what it is. Have fun! :)

Python Basic Data Structures and Common Commands Cheatsheet

Lists	[]	list	append, extend, pop, index
Tuples (immutable)	() or {},	tuple
Sets	{'',}
Dictionary	{}	dict

List | Common Commands:

Append | For adding a single element. If you give it a list, it'll add a list of a list

Extend | for merging a lot of elements

Clear | for clearing the element

Index | for searching the position of an item, first occurrence.

Remove | for removing a specific item, first occurrence

pop | for removing a single item from at a INDEX position (AND returning it if need be). 

Copy | For creating a new object. Anyway else will just reference the old object

Reverse | Flips things over

Count | Built in Frequency Counter

Examples

Iterating in Python Sampler

for i in [1, 2, 3]:

print (i)

for ch in "Yay!":

print(ch)

for num in range(5):

print('David is good at this!')

help(range)

phrase="Don't Panic!"

All the Letters

phrase

Every Third Letter up to index location 10

phrase[0:10:3]

All letters up to but not including the 10th

phrase[:10]

Only the first three letters

phrase[3:]

Every Second Letter

phrase[::2]

How to count backwards 1

backwords = phrase[::-1}

''.join(backwords)

>>> phrase[::-1]

"!cinap t'noD"

How to count backwards 2

>>> phrase = "The Sky is Blue"

>>> words = phrase.split(' ')

>>> words

['The', 'Sky', 'is', 'Blue']

>>> words[::-1]

['Blue', 'is', 'Sky', 'The']

Spring Core Containers and Dependency Injection

Java classes should be as independent as possible from each other.

• To decouple classes from one another, dependencies should be injected through Constructors and Setters.
• Classes should not configure themselves. IoC uses dependency injection to: 
• Configure a class correctlyi from outside the class
• Wire services or components.
• Piecing together all the beans in the Spring Container is called Wiring.
• This can most commonly be don through xml.
• Various BeanFactories and ApplicationContext objects that support wiring include:
• XMLBeanFactory
• ClassPathXMLApplicationContext
• FileSystemXMLApplicationContext
• XMLWebApplicationContext

Spring Fundamentals

Originally developed to fight against low level maintenance, Spring framework functionality can be used in any JavaEE server and most of it is also adaptable to non-manage frameworks.

Most JavaEE applications are complex and require a lot of effort to develop.

Spring's BeanFactory enables configuration options from XML by providing dependencies when required, automatically.

Spring is composed of 7 key components. These modules can be chosen freely as needed.

1. Spring Core - separates configuration from dependencies
• Prime Components: BeanFactory; Application contexts.
• Bean Factory: Provides support for dependency injection.
• Application Contexts: provides application framework services.
2. Spring Context
• A configuration file that provides contextual information to the Spring Framework. JNDI, EJB, email, scheduling and internalization services.
3. Spring AOP
• Integrates Aspect Oriented Programming through its configuration management feature, providing transaction management services for objects in any Spring-based application.
4. Spring DAO
• Data Access Object abstraction layer offers a meaningful hierarchy for the managing of exceptions and error messages. This greatly reduces the amount of code you need to write, such as opening and closing connections.
5. Spring ORM
• Object Relational Map -Object oriented programs and relational databases tend to speak completely different languages. This bridges that gap. This provides support to several ORM solutions, including Hibernate, JDO, JPA, and others.
6. Spring Web Module
• Builds on top of the application context module, to provide contexts for Web-based applications. The Web module
7. Spring MVC
• A full featured MVC implementation for building Web-applications. Highly configurable via strategy interfaces. It accommodates numerous view technologies including JSP, Velocity, Tile, and so on.

Git Tips

Git can be confusing. You try cloning someone else's repo and it doesn't work. Or you = try to move branches…its all kinda counter intuitive. Unless I write down some important lessons learned right here.

Add Collaborators

• The master of the repository has to add users to allow them to interact with stuff. I'm yet to master the security features, as I accidentally overrode the master branch today (DAMMIT)
• Here is the link to the guide used.

Who are you?

To collaborate you must set your glob= al user variables under config.

• git config --global user.= name
• git config --global user.email

1. git con= fig --global core.editor [vi | cat | w]

   Change = the main editor to vi or cat, etc

Git Branch =Making New Backup Folders

I like to think of branches as different backup folders. As one familiar with most DOS commands, what is the equivalent of making a new folder (mkdir) and cding into it? Let along listing (ls) them?

mkdir	git branch [branchName]
ls	git branch -= a
cd	git checkout [branchName]

1. git branch -a =3D=3D ls [= your branches] 
2. git branch [branchName]= =3D=3D create a new branch
3. git checkout [branchNam= e] =3D=3D cd to your brannch

Git Add = Prepare to Save.

1.  You need to use -a or . from within t= he same folder as the .gitignore command for this to work.

Git Commit = Save Local Copy, but you need to have a message (apparently)

• Commit doesn't like you unless you add in -m after calling the command. So you must either vi or cat or something else. As long as there is a message.

After Cloning, you have to work within the folder itself

• No amount of trying to add the folder I cloned seemed to cause the thing to be sta= ged. You have to work where the .gitignore file is.

Resizing Disks in Virtual Box

Use this guide

Quick Tips:

1. This will only work on Dynamically Allocated Disks. It will NOT work on fixed disks.
2. Put .\ in front of the command vboxmanage.exe

Sample Command:

.\vboxmanage.exe modifymed= ium "C:\Users\David\VirtualBox VMs\Ubuntu2019\VirtSSD.vhd" --resize 3= 0999

Virtual Box Optimization 2

Lately, I've grown very serious about the power of Linux.

I experimented with a couple of guides on how to optimize your virtual Linux experience and discovered that a workable environment is indeed doable. Unfortunately, I allocated my first successful Virtual Box to have a very small hard disk of ten gigabytes. So, I experimented with simply reinstalling the entire thing, only to suffer from additional issues.

Here, I roughly jot down lessons learned.

Environment:
Windows 8 -> Windows 10
C Drive: SSHD
D Drive: HHD
E Drive: SSD

Experiment #1: Installing Virtual Hard Disk onto a machine separate from the host Machine.

It is said that installing a virtual hard drive onto a separate (actual) disk (such as on disk D) would help spare resources. Unfortunately, it only appeared to lead to other issues. Though it installed at first, eventually, there were hang ups as it refused to start. Guides that I looked into implied that there are in fact, issues concerning having Virtual Box look for a VHD that is on another

Experiment #2: Reinstalling Virtual Box onto the external drive.

As such, I decided to do several experiments with a new SSD that I purchased for pure storage. I reinstalled VirtualBox to my E drive, but now all Virtual Machines refuse to start, including new virtual machines I make from scratch. Seems this blogger has a lot more lessons to learn, and will share them in time.

Virtual Box Optimization

If you are a developer who works in windows, it is nonetheless important to know and work with linux. If you're like me, most of your linux experience has been via VirtualBox, where you followed a few guides and installed it.

If you're like me, you might have then mistakenly begun to associate linux with "slow."

However, what if I told you that if you think a linux virtual machine is bad, then the problem is with your Virtual Configurations and not with the operating system.

Allow me to share a couple guides and configuration tips.

Key Takeaways

I. Ensure that you are using as many processors as you can spare.

[pict pending]

Don't just leave things at the default processor size. Give it 4 cores if at all possible.

II. Invest in more RAM and assign (at least) 8 GB of Ram to your Virtual Machine

[pict pending]

RAM is the easiest piece of hardware to swap. Though it might seem expensive, investing in 32 GB of RAM is a godsend to your host environment if you also want it to run a true virtual environment alongside it.

Upgrade your RAM and ensure you are using a fair amount of it for your VM.

III. Ensure the Display is set to 128 GB and tied to a graphics card (if applicable)

[pict pending]

As with videogames, weak graphical memory leads to weak performance and LAG spikes. If you want decent FPS (frames per second) with VirtualBox, you have to treat it as you would a video game.

If you are using NVidia GPU, open the NVidia Control Panel and have it override the default settings for VirtualBox.

In conjunction with this, make sure the VBoxVGA is chosen as your graphics controller. Keep Acceleration disabled.

Additional Resources

• https://www.techrepublic.com/article/how-to-improve-virtualbox-guest-performance-in-five-steps/
• https://www.tecmint.com/install-virtualbox-guest-additions-in-ubuntu/

Database Principles - Keeping Things Normal

Lately, I keep busy by acting as web administrator to a website. Its important however to keep your data categorized. Here are some tips on how to do so...

Atomicity
Consistency
Isolation
Durability

These are the basics, but I don't have time to elaborate now crap dammit (to be continued)

MVC and the Web

Here are some brief notes on MVC and the Web taken from "Design Patterns" by O'Reilly publishing.

1) You make an HTTP request, which is received by a servlet

Using your web browser, you make an HTTP request (usually with some data, like a user name and password). A servlet receives this form of data and parses it.

2) The Servlet acts as the controller

The servlet plays the role of the controller and processes your requests, most likely making requests on the model (usually a database). The result of processing the request is usually bundled up in the form of a JavaBean.

3) The controller forwards control to the view

The View is represented by a JSP. The JSP's only job is to generate the page representing the view of the model (via a Javabean) along with any controls needed for further actions.

4) The view returns a page to the browser via HTTP

A page is returned to the browser, where it is displayed as the view. The user submits further requests, which are processed in the same fashion.

A Java Server Page:

•Enables development of a dynamic user interface

•Is an extension of a Servlet

•Is a combination of:

–Snippets of Java code

–Hyper Text Markup Language (HTML)

–JavaScript

–Cascading Style Sheets (CSS)

--What is a Design Pattern? If not a solution to a problem in a context? (O'Reilly pg. 579)

Model View Controller

Model

•Knows about the data that needs to be displayed.

•Knows about the operations that can be applied on the data.

•Does not know how to display the data to the user. 

View

Really just the observer pattern.

•Provides Graphical User Interface components

•Relays user requests

•Uses the query methods

•Displays information

•Maintains consistency

Controller

•Translates interactions with the view into actions.

•Is responsible for mapping end-user action to application response.

Hashing

Hashing is all about adding integrity to information.

The Hash takes a fixed amount of data and maps it to a new piece of data of fixed sized.

It doesn't matter how big the source data is, the hash will be the same size..

Hashes are NOT encrypted versions of the original texts.

Properties:

1. Hashes are one way. You can't reverse the hash.
2. They are deterministic.
1. One letter change leads to a completely different result. 

Common Uses:

1. Verify a download file
1. Compare the downloaded file hash with the posted hash value. So if the file is corrupt (because a SINGLE byte of data is missing), we know something is wrong.
2. Passwords
1. Since two different messages have a completely different output, even if a single character changes, they are great for passwords. Why store the password when you can store the hashes and compare, that way people who might see the hash CANNOT guess your plaintext password.
3. Digital Signatures
1. Prove that the message sent was not intercepted and changed.
2. We know that the digitial signature is not fake by ENCRYPTING the HASH of with the private key.  This is the digital signature.
1. The sig is then sent WITH the message.
2. The recipient then hashes the message for himself, and then DECRYPT the digital signature hash of with the public key and if the hashes match, we know that the message was not tampered with. hashing the private key.

Anyway, here's a table speaking to different types of hashs. One of the key things to know is that primitive hashes are vulnerable against collisions, and with that, become easier to crack. Several inputs can lead to the same output in the case of weak hashing algorithms.

MD5 (Message Digest v. 5)	Grandpa of Hashes (1992), 128 bit	128
SHA (Secure Hash Algorithm)	Family of Hashes developed by National Institute of Standards	168 Bit v 1. 512 has never had a collision though.
RIPEMD	128, 160, 256 and 360 versions

Go ahead and play with different hashes here. 

Or check the security of your password somewhere like here.

Webcrawling

Webcrawling.

 Sometimes you just need to grab a bunch of text from a website really quick and fast. This is where its nice to have handy a few web-crawling algorithms for some extreme copy and paste. I once had to quickly gather all the headlines from a website and there was just no time to copy and paste it all. So I imported BeautifulSoup and just skimmed for the headers of each application.

• Step 1) Import BeautifulSoup and the request packages. 
• Step 2) Make a request to your target website and safe the raw html document. 
• Step 3) Have your BeautifulSoup object skim through whatever it is you need to grab.

 Key Code to remember:  

from bs4 import BeautifulSoup 
import re '''This is the request package for making httprequests''' 
html_doc=""" """ 

def extractHeaders(string): 
    soup=BeautifulSoup(string, 'html.parser') 
        for tag in soup.find_all(re.compile("h4")): 
        print(tag)

def extractHeaders(string, tag): 
    soup=BeautifulSoup(string, 'html.parser')
        for tag in soup.find_all(re.compile(tag)): print(tag)

Key takeaway: Search by the html tag you want targeted, like 'h4' or 'title' or something.

I noticed that in this case however, its important to make your request <i>seem</i> like it is from an actual web browser. The headers parameter of the request can handle this. To be certain, set your headers as something that follows below.

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0'}

Then try running your request with that header as your header.

r= requests.get('target_html', headers=headers)