How to edit table so that index and column names are on the same row. import beatifulsoup. from bs4 import BeautifulSoup find the empty tags in ‘thead’ and then fill with col names from the dataframe. Note: If you code “header=False”, the default style for table row will be gone <tr style=”text-align: right;”> html_doc = tide_table.to_html(header=False) soup = BeautifulSoup(html_doc, ‘html.parser’) […]
Pandas – how to edit columns for html table
Pandas/Python – how to drop a row with ‘NaN’
So, you want to get rid of any row that has ‘NaN’ ( null or not a number ) values, because it doesn’t work with some functions ( or can’t ignore ) obs1.dropna(how = ‘all’, subset = [‘wind_mph’], inplace = True) obs1 = obs1.reset_index(drop=True) Another important point – is that with Pandas – you have to convert any value you […]
Using git to reset file
So, you were editing a file on production machine to debug an issue. Once you are finished you want to reset it back to what was in git. delete the file. 2. git checkout HEAD <filename> This will restore the file to the last pull or fetch. v( local git repository ) However, when you pull again. it will show […]
Windows – how to start Jupyter notebook without anaconda
So, you pip installed Jupyter. Thinking – hey don’t have to start up Anaconda environment to run it. C:\Users\jonallen\Documents\github\weather_obs>pip install jupyter Collecting jupyter Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB) Collecting ipywidgets Downloading ipywidgets-7.6.3-py2.py3-none-any.whl (121 kB) |████████████████████████████████| 121 kB 3.3 MB/s Collecting notebook Downloading notebook-6.4.3-py3-none-any.whl (9.9 MB) |████████████████████████████████| 9.9 MB 234 kB/s Collecting jupyter-console Downloading jupyter_console-6.4.0-py3-none-any.whl (22 kB) Collecting nbconvert Downloading nbconvert-6.1.0-py3-none-any.whl (551 […]
Using Pandas append on groups of CSV files
This function loads a series of daily csv files and combines them into a monthly dataframe Create an empty Pandas DataFrame append to empty DataFrame after each load. Word of caution – the columns must be the same for all inputs. One should test the columns to make sure you get what you expect. def load_monthly_noaa_csv_files( dir, noaa_station, ext = […]
Using assert for simple test harness
Here is an example of using assert. This can be found in my fork on github – Fork of PySitemap. The function returns True or False ########################################### # test harness for function wp_remove_dup ########################################### from main import wp_remove_dup test_list = [“https://www.jibsheet.net/linux”, \ “https://www.jibsheet.net/linux/index.php/2021/07/07”,\ “https://www.jibsheet.net/linux/index.php/2021/”,\ “https://www.jibsheet.net/linux/index.php/page/3/”,\ “https://www.jibsheet.net/linux/index.php/2021/05/08/”, “https://www.jibsheet.net/linux/index.php/tag/linux/”] print(test_list) for item in test_list: print(wp_remove_dup(item)) assert (wp_remove_dup(test_list[0]) == False) assert (wp_remove_dup(test_list[1]) […]
How to check for bad filename or path in BASH
So, I am reading about injection attacks. White Listing is better than Black Listing. Bash support some regex. The first IF – check for a valid file name. 2nd checks for a Unix path name and allows spaces. But not “&, >, <, or | “. Ampersand in Linux is really bad. if you allow that in certain user inputs […]
Python How to reference class functions in class variable.
In static languages – it makes since to put the class variable at the top of the Class. However, since Python is dynamic, it can’t know about the functions until it processes them. class AwsProcessor: service_dict = {‘s3’: process_s3, ‘ebs’: process_ebs} def __init__(self, processing_service): self.service = processing_service def process_ebs(self): pass def process_s3(self): pass This gives error File “c:\aws-scripts\class AwsProcessor.py”, line […]
How to automatically generate requirements.txt for python projects
pip install pipreqs pipreqs /path/to/project stack exchange discussion If you have multiple sub directories – one should view the comments on the stack exchange above.
Graviton – no mail client
Here is how to install the mail client on AWS Graviton EC2 instance Note: mutt not available. You can install Alpine. I think you will have to find a mutt rpm online if you really want mutt. [ec2-user@graviton_host]$ mail -bash: mail: command not found [ec2-user@ip-172-31-67-136 weather_obs]$ sudo bash [root@graviton_host]# find / -name mail /var/mail /var/spool/mail /usr/local/lib/python3.7/site-packages/jedi/third_party/django-stubs/django-stubs/core/mail [root@graviton_host]# exit exit […]