Blog

  • AccountServiceApp

    Account Service App

    This is an implementation of Spring Security to provide a simple database backed authentication method that is intended to run as a remote service such as an EJB. It was developed to provide a common service for all my web applications to authenticate user access through database configuration. The interface exposes a login using the Spring AbstractAuthenticationToken request as well as application specific methods to authenticate an application and create a login. A second login method is provided to use an application provided login request format.

    Authentication is carried out using the Spring ProviderManager which is configured in a similar fashion to the following example. A full test configuration is available here

    <bean id="authenticationManager" class="org.springframework.security.authentication.ProviderManager">
      <property name="providers">
        <list>
          <ref local="daoAuthenticationProvider"/>
          <ref local="anonymousAuthenticationProvider"/>
          <ref local="ldapAuthenticationProvider"/>
        </list>
      </property>
    </bean>
    
    <bean id="daoAuthenticationProvider" class="org.springframework.security.authentication.dao.DaoAuthenticationProvider">
      <property name="userDetailsService" ref="inMemoryDaoImpl"/>
      <property name="passwordEncoder" ref="passwordEncoder"/>
    </bean>
    

    Encryption uses bcryprt through the supplied BCryptPasswordEncoder. Other functions are available in the current Spring Security release.


    For convenience the project incorporates both the definition of the data model requirements and an implementation using Hibernate for a relational database design. In normal circumstances these would be separate projects and this may be easily achieved by separating out the relevant Hibernate package if required.

    The data model is quite simple and naturally open to enhancements but suffices as a starting point. The following diagram shows that it basically implements a many-to-many relationship between users and accounts. Access can be controlled at any of the AccountUser, Account or Application levels and the Application can be closed to new sign-ups if required.

    Data model

    This project does not provide an interface into maintaining the data model but does implement a simple DAO pattern which provides many of the methods for this purpose. These are used in the test files which carry out the basic CRUD actions. Testing uses the HSQLDB in-memory database which means that no RDBMS needs to be installed but if DDL is required then the following is the Hibernate output (constraint names slightly amended) may be used to create a MySQL database.

    create table access_level (id integer not null auto_increment, description varchar(255), primary key (id));
    
    create table account (id integer not null auto_increment, active char(1) not null, application_id integer not null, create_date datetime, resource_name varchar(255), primary key (id));
    
    create table account_user (user_id integer not null, account_id integer not null, access_level_id integer not null, account_message varchar(255), create_date datetime, enabled bit not null, last_access_date datetime, primary key (user_id, account_id));
    
    create table application (id integer not null auto_increment, enabled bit not null, name varchar(255), registrationOpen bit not null, primary key (id));
    
    create table user (id integer not null auto_increment, create_date datetime, enabled bit not null, password_salt tinyblob, user_key varchar(255), user_name varchar(255), user_password tinyblob, primary key (id));
    
    alter table application add constraint UK_APPLICATION_NAME  unique (name);
    alter table user add constraint UK_USER_USER_KEY  unique (user_key);
    
    alter table account add constraint FK_ACCOUNT_APPLICATION_ID_TO_APPLICTION_ID foreign key (application_id) references application (id);
    alter table account_user add constraint FK_ACCOUNT_USER_ACCOUNT_TO_ID_ACCOUNT_ID foreign key (account_id) references account (id);
    alter table account_user add constraint FK_ACCOUNT_USER_USER_ID_TO_USER_ID foreign key (user_id) references user (id);
    alter table account_user add constraint FK_ACCOUNT_USER_ACCESS_LEVEL_ID_TO_ACCESS_LEVEL_ID foreign key (access_level_id) references access_level (id);
    

    Visit original content creator repository

  • simple-receptive-field

    simple Numerical Receptive Fields

    Analyzing the Receptive Field for a Convolutional Neural Network can be very useful in debugging and/or better understanding of how the model’s output looked at the input.

    The RF can be mathematically derived (a good blogpost on receptive field arithmetic and this excellent distill.pub). But we can also take advantage of automatic differentiation libraries to compute the RF numerically.

    Steps to compute RF

    1. Build the dynamic computational graph of the conv block
    2. Replace output gradients with all 0s
    3. Pick a (h, w) position in this new gradient tensor and set it to non-zeros
    4. Backprop this gradient through the graph
    5. Take the .grad of the input after the backward pass, and look for non-zero entries

    Usage

    Can refer to the demo notebook or:

    import torch
    from numeric_rf import NumericRF
    
    # ... given an image tensor `im`
    
    convs = torch.nn.Sequential(
                                    torch.nn.Conv2d(3, 16, (5,3), stride=(3,2)),
                                    torch.nn.Conv2d(16, 16, (5,3), stride=2),
                                    torch.nn.Conv2d(16, 16, 3, stride=2),
                                    torch.nn.Conv2d(16, 16, 3, padding=1),
                                    torch.nn.Conv2d(16, 8, 3),
            )
    
    rf = NumericRF(model = convs, input_shape =  im.shape)
    
    rf.heatmap(pos = (4, 8))
    
    rf.info()
    
    rf.plot(image = im, add_text = True)

    Will give both the receptive field for that output position:

    {
    	'h': {
    		'bounds' : (18, 58), 
    		'range'  : 40}, 
    		
    	'w': {
    		'bounds' : (28, 50), 
    		'range'  : 22}
    }
    

    And also the visualization:

    Verification

    A quick way to verify that this approach works can be found in the demo notebook, by following these steps:

    1. Calculate the receptive field for the conv block at a given position (important to zero out the bias term!)
    2. Create an input shaped tensor of 1s
    3. Zero out all input entries that fall in the Receptive Field
    4. After a forward pass, the only zero entry should be the chosen position (other factors may contribute to zeros in the output, e.g. padding, but the initial output position must be 0)

    Illustrated here:

    Visit original content creator repository
  • Malastare.ai

    Building a startup that is focused exclusively on data science projects in two hypergrowth industries, blockchain, and artificial intelligence is truly exciting and game-changing. The reality is Malastare is a hypergrowth startup and it just isn’t for everyone. Unapologetically, we work hard, run hard, keep bureaucracy to a minimum and hold ourselves accountable in our quest to build something special. If these things interest you, you are in the right place.

    We are a team of product visionaries, talented engineers, data scientists, and customer-obsessed inventors who tackle the world’s most complex deep learning, artificial intelligence challenges in order to deliver best-in-class infrastructure and software solutions.

    Our cutting-edge engineers and data scientists create disruptive real-world applications to realize the promise of nascent blockchain and AI technologies and provide highly secure and feature-rich solutions for our customers. We leverage the most advanced deep learning neural networks, based on best-in-class sentiment analysis and market analytics.

    We are a startup with a strong impetus to innovate, drive end to end ownership and meet critical business goals. You get to rub shoulders daily with outstanding software, hardware and embedded engineers, data scientists, UX designers and the executive leadership team in an environment of high-speed learning and professional development. Our employees thrive in a unique team environment where energy, creativity, and collaboration drive innovation.

    Our future with AI

    Our expertise covers the following:

    Business Logistics, Including Supply Chain Optimization

    Supply chain plays an important role in tracking the goods and services that a company manufactures. This asset of the company was initially undertaken with the help of management tools until the advent of analytics. Analytics uses the historical enterprise data to train predictive models in machine learning that make informed business decisions. It enables one to identify inefficiencies to capture greater cost savings. Sometimes, risk modeling is conducted to do a “pre-mortem” for future threats. Analytics are also useful in identifying the key factors that enable better functioning of supply chain management.

    FinTech Industries

    After the recession in 2008 hit the financial sector and left everyone panting for more, the importance of Big Data in finance was realized. The BFSI (Banking and Finance Services Industry) came up with many sorts of data and along with many sorts of problems which were only possible to solve with the help of big data.

    The three Vs of big data – velocity, volume and variety have accelerated the banking sector in various aspects. In investment banking, the volume of data is growing manifold day-by-day and so has its velocity. Cloud computing and Hadoop have enabled the big data platforms to cater to the tremendous volume of data. Banks have now realized the potential of big data and are employing the same in their systems to make it future-proof.

    Health And Wellness

    The evolution of the healthcare domain was only observed after big healthcare organizations started investing in big data. The amount at which the data was flowing in, made it difficult for the traditional data-processing application software to cope with the data. Big data has tools like Hadoop, MapReduce, Hive, Pig which help in proper management of big volumes of data as well as in processing it for the welfare of the domain. At times, cancer is untraceable in a human body which leads to late detection or no detection at all. Big data helps in solving that problem.

    Patients need their body-metrics tracked every time which involves everything from heartbeat to sugar levels. Big data takes care of that too.

    Education And Electronic Teaching

    Online teaching and education are taking the market by storm as it is been adopted by more and more organizations. Online courses which can be accessed from anywhere are being recommended. The recommender system in data science is implemented for this purpose and assigns courses as per one`s interests.

    Climate And Energy

    Weather forecasting is the most common application and it has been operating for quite a long time. Predicting any weather changes like storms, thunder, heavy rainfall makes it easy for forecasters to analyze and understand the future scenario much better. Issues like carbon emissions and hike in the level of the sea due to glaciers, environmental changes, and ecological disbalances can be calculated before its advent have any catastrophic effects on nature and human lives.

    Data Processing Overview

    Python Workflow

    To carry out data cleansing, data transformation, statistical modeling, data visualization, machine learning, and countless more.

    RStudio Workflow

    To perform operations on data.

    R takes care of visualization by providing a shiny tool. RShiny takes care of interactive web applications for visualizing data which brings data analysis in R to life.

    Apache Spark

    Used to execute projects in Scala, Java, SQL, Python, or R. Considered as a unified engine which is used to take analytical decisions from large-scale data processing.

    For managing clusters, Spark provides standalone Hadoop YARN. For distributed storage, Spark offers an interface with a wide variety of applications like Hadoop Distributed File System, MapReduce File System, Cassandra.

    Microsoft Excel

    To carry out calculations, visualize graphs.

    SQL

    SQL programming language used to handle and manipulate structured queries stored in relational database systems also know as RDBMS.

    Tableau

    Tableau data visualization tool used for representing data in terms of charts and dashboards open and generate useful insights for clients. Its capability to handle relational databases, OLAP cubes, spreadsheets and also generate a number of graph types depending on the type of data retrieved. The latitude and longitudes features of a location offered in Tableau creates a geographic representation of any reports regarding sales, profit or any other factors which can be represented with the help of maps.

    Power BI

    Power BI as a business intelligence tool developed by Microsoft. It provides interactive visualizations coupled with BI capabilities, where we build our own customized reports and dashboards, without having to depend on information technology users or database administrators.

    Its cloud-based BI services enables connecting to hundreds of data sources in the cloud, using power query to simplify data ingestion, transformation, integration, and enrichment.

    Our Problem Solving Roadmap

    License

    These project are licensed under the MIT License – see the LICENSE.md file for details

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

    Visit original content creator repository
  • Mask_RCNN

    Mask R-CNN for Object Detection and Segmentation

    This is an implementation of Mask R-CNN on Python 3, Keras, and TensorFlow. The model generates bounding boxes and segmentation masks for each instance of an object in the image. It’s based on Feature Pyramid Network (FPN) and a ResNet101 backbone. (Modified from the actual implementation https://github.com/matterport/Mask_RCNN)

    The repository includes:

    • Source code of Mask R-CNN built on FPN and ResNet101.
    • Training code for MS COCO
    • Jupyter notebooks to visualize the detection pipeline at every step
    • ParallelModel class for multi-GPU training
    • Evaluation on MS COCO metrics (AP)
    • Example of training on your own dataset

    Getting Started

    • demo.ipynb Is the easiest way to start. It shows an example of using a model pre-trained on MS COCO to segment objects in your own images. It includes code to run object detection and instance segmentation on arbitrary images.

    • train_shapes.ipynb shows how to train Mask R-CNN on your own dataset. This notebook introduces a toy dataset (Shapes) to demonstrate training on a new dataset.

    • (model.py, utils.py, config.py): These files contain the main Mask RCNN implementation.

    • inspect_data.ipynb. This notebook visualizes the different pre-processing steps to prepare the training data.

    • inspect_model.ipynb This notebook goes in depth into the steps performed to detect and segment objects. It provides visualizations of every step of the pipeline.

    • inspect_weights.ipynb This notebooks inspects the weights of a trained model and looks for anomalies and odd patterns.

    Step by Step Detection

    To help with debugging and understanding the model, there are 3 notebooks (inspect_data.ipynb, inspect_model.ipynb, inspect_weights.ipynb) that provide a lot of visualizations and allow running the model step by step to inspect the output at each point. Here are a few examples:

    1. Anchor sorting and filtering

    Visualizes every step of the first stage Region Proposal Network and displays positive and negative anchors along with anchor box refinement.

    2. Bounding Box Refinement

    This is an example of final detection boxes (dotted lines) and the refinement applied to them (solid lines) in the second stage.

    3. Mask Generation

    Examples of generated masks. These then get scaled and placed on the image in the right location.

    4.Layer activations

    Often it’s useful to inspect the activations at different layers to look for signs of trouble (all zeros or random noise).

    5. Weight Histograms

    Another useful debugging tool is to inspect the weight histograms. These are included in the inspect_weights.ipynb notebook.

    6. Logging to TensorBoard

    TensorBoard is another great debugging and visualization tool. The model is configured to log losses and save weights at the end of every epoch.

    6. Composing the different pieces into a final result

    Training on MS COCO

    We’re providing pre-trained weights for MS COCO to make it easier to start. You can use those weights as a starting point to train your own variation on the network. Training and evaluation code is in samples/coco/coco.py. You can import this module in Jupyter notebook (see the provided notebooks for examples) or you can run it directly from the command line as such:

    # Train a new model starting from pre-trained COCO weights
    python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=coco
    
    # Train a new model starting from ImageNet weights
    python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=imagenet
    
    # Continue training a model that you had trained earlier
    python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=/path/to/weights.h5
    
    # Continue training the last model you trained. This will find
    # the last trained weights in the model directory.
    python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=last
    

    You can also run the COCO evaluation code with:

    # Run COCO evaluation on the last trained model
    python3 samples/coco/coco.py evaluate --dataset=/path/to/coco/ --model=last
    

    The training schedule, learning rate, and other parameters should be set in samples/coco/coco.py.

    Training on Your Own Dataset

    Start by reading this blog post about the balloon color splash sample. It covers the process starting from annotating images to training to using the results in a sample application.

    In summary, to train the model on your own dataset you’ll need to extend two classes:

    Config This class contains the default configuration. Subclass it and modify the attributes you need to change.

    Dataset This class provides a consistent way to work with any dataset. It allows you to use new datasets for training without having to change the code of the model. It also supports loading multiple datasets at the same time, which is useful if the objects you want to detect are not all available in one dataset.

    See examples in samples/shapes/train_shapes.ipynb, samples/coco/coco.py, samples/balloon/balloon.py, and samples/nucleus/nucleus.py.

    Differences from the Official Paper

    This implementation follows the Mask RCNN paper for the most part, but there are a few cases where we deviated in favor of code simplicity and generalization. These are some of the differences we’re aware of. If you encounter other differences, please do let us know.

    • Image Resizing: To support training multiple images per batch we resize all images to the same size. For example, 1024x1024px on MS COCO. We preserve the aspect ratio, so if an image is not square we pad it with zeros. In the paper the resizing is done such that the smallest side is 800px and the largest is trimmed at 1000px.

    • Bounding Boxes: Some datasets provide bounding boxes and some provide masks only. To support training on multiple datasets we opted to ignore the bounding boxes that come with the dataset and generate them on the fly instead. We pick the smallest box that encapsulates all the pixels of the mask as the bounding box. This simplifies the implementation and also makes it easy to apply image augmentations that would otherwise be harder to apply to bounding boxes, such as image rotation.

      To validate this approach, we compared our computed bounding boxes to those provided by the COCO dataset. We found that ~2% of bounding boxes differed by 1px or more, ~0.05% differed by 5px or more, and only 0.01% differed by 10px or more.

    • Learning Rate: The paper uses a learning rate of 0.02, but we found that to be too high, and often causes the weights to explode, especially when using a small batch size. It might be related to differences between how Caffe and TensorFlow compute gradients (sum vs mean across batches and GPUs). Or, maybe the official model uses gradient clipping to avoid this issue. We do use gradient clipping, but don’t set it too aggressively. We found that smaller learning rates converge faster anyway so we go with that.

    Requirements

    Python 3.4, TensorFlow 1.3, Keras 2.0.8 and other common packages listed in requirements.txt.

    MS COCO Requirements:

    To train or test on MS COCO, you’ll also need:

    If you use Docker, the code has been verified to work on this Docker container.

    Installation

    1. Clone this repository

    2. Install dependencies

      pip3 install -r requirements.txt
    3. Run setup from the repository root directory

      python3 setup.py install
    4. Download pre-trained COCO weights (mask_rcnn_coco.h5) from the releases page.

    5. (Optional) To train or test on MS COCO install pycocotools from one of these repos. They are forks of the original pycocotools with fixes for Python3 and Windows (the official repo doesn’t seem to be active anymore).

    Visit original content creator repository
  • boat

    Boat

    This project is a single-page landing site dedicated to promoting boat trips, yachting, and rental services. The site is designed to provide a visually appealing and informative experience for users interested in maritime adventures. Below is a detailed description of each section of the website.

    Table of content

    Routing

    The app uses react-scroll for navigation. The available sections are:

    • Home – Welcome section includes:

    Navigation: A responsive navigation menu that adapts seamlessly to mobile devices, ensuring easy access to all sections of the site. Main Banner: Features an eye-catching image of a motorboat riding the waves of the ocean, setting the tone for the adventurous theme of the site.

    • Speedy – includes:

    Yacht Display: Showcases images of various yachts, highlighting different options available for rent. Animated Card: An engaging, animated card providing detailed information and options for renting a vessel, making the user experience interactive and dynamic.

    • Safety – includes:

    Importance of Safety: This section emphasizes the critical importance of safety while boating, ensuring users understand the measures taken to protect them during their maritime adventures.

    • Gallery – includes:

    Photo Slider: A gallery presented in a slider format, displaying beautiful and captivating images of boats, yachts, and ocean scenes to inspire and attract potential customers.

    • Contact – includes:

    Contact Form: A simple and effective form for users to get in touch, ask questions, or request further information about the services offered.

    Features

    Implemented responsiveness and mobile menu for better UI.

    Getting Started

    To run the application locally, follow these steps:

    1. Clone the repository: git clone https://github.com/Inna-Mykytiuk/boat.git
    2. Install dependencies: npm install
    3. Run the app: npm run dev
    4. Open http://localhost:3000 in your browser (Note: the port may be changed if 3000 port is occupied by another website).

    Technologies Used

    1. Next.js: Our website is powered by Next.js, providing a seamless and efficient user experience. It ensures fast loading times and smooth navigation.

    2. Tailwind CSS: Used for styling, offering a highly customizable and utility-first approach to CSS that ensures responsive and attractive design.

    3. React-Scroll: Enhancing the scrolling experience on our website, React-Scroll adds a touch of sophistication, allowing users to glide smoothly between sections.

    4. TypeScript: Implemented for static type checking, enhancing code quality and reducing errors during development.

    5. Framer Motion: Our project incorporates Framer Motion to bring life to the user interface with stunning animations. Framer Motion is a React library that simplifies the creation of smooth and interactive animations, adding a layer of dynamism to various elements on our website.

    6. Swiper: Used for creating the responsive, touch-friendly photo slider in the Gallery section, providing a seamless viewing experience on all devices..

    Summary

    This landing page effectively combines visual appeal with functional design to create an engaging user experience for individuals interested in boat trips and yacht rentals. By leveraging modern web development technologies, the site ensures high performance, responsiveness, and an interactive user interface. The strategic use of animations, smooth scrolling, and an intuitive navigation system further enhances the overall experience, making it easy for users to explore the services offered and get in touch for more information.

    preview preview

    Visit original content creator repository
  • Leaflet.MoveMarker

    General Information

    L.MoveMarker is a leaflet plugin that is used to create moving marker and also trail polylines animation, equipped with a variety of features that are quite complete. plugin currently built in leaflet version ^1.9.2.

    Feature

    Marker Polyline
    animate marker animate polyline
    duration by speed duration by speed ❌
    rotate marker show/hide polyline
    show/hide marker limit max polyline length
    disable/enable animate marker disable/enable animate polyline
    stop animate marker stop animate polyline
    follow marker when moving

    How to install

    you can go to npm package

    npm i l.movemarker

    Import

    if you are using ES6, you can import

    import 'l.movemarker'

    Demo

    demo description
    basic demo (codesandbox) basic demo is a simple usage example
    basic demo (github) basic demo is a simple usage example
    real case demo an example of a real L.MoveMarker implementation using multiple markers and the lat lng mock API

    First Call

    prevLatLng = [lat, lng] (required) first position
    nextLatLng = [lat, lng] (optional) it will be call animate to next position
    if you just fill lat lng without nextLatLng, it will still work and will only create marker that are positioned in prevLatLng

    const instance = L.MoveMarker(
      [[prevLatLng], [nextLatLng]],
      polylineOptions,
      markerOptions,
      featureGroupOtions
    )

    the L.MoveMarker will return the instance layer L.FeatureGroup and it inside have many layers L.MotionLine and one layer L.MotionMarker.

    Options

    Polyline Options

    props type default value description
    color string red used to set the color of the polylines
    weight number 4 used to set the weight of the polylines
    animate boolean true this will activate the moving animation effect on the polyline
    duration number 5000 value in milliseconds, we recommended duration the marker and the polyline must same
    hidePolylines boolean false hide polylines from map, you can also show/hide with method hidePolylines(boolean), for more detail see on method documentation
    removeFirstLines boolean false this will remove first line of polylines, this prop have relation with prop maxLengthLines
    maxLengthLines number 3 This prop used to limit the number of polylines and if it exceeds the limit then the first polyline will be deleted

    You can also fill in the default options polyline from the leaflet polyline options polyline

    Marker Options

    props type default value description
    animate boolean true this will activate the moving animation effect on the marker
    duration number 5000 value in milliseconds, we recommended duration the marker and the polyline must same
    followMarker boolean false this will activate the follow marker feature, when the marker moves, the screen will follow the movement of the marker
    hideMarker boolean true hide marker from map, you can also show/hide with method hideMarker(boolean), for more detail see on method documentation
    rotateMarker boolean false this will activate the rotation marker and props rotateAngle will be required
    rotateAngle number 0 0 to 360 degrees
    speed number 0 if the speed value is filled then the duration props will not work and the value unit is km/h

    You can also fill in the default options marker from the leaflet marker options marker

    Feature Group Options

    currently we dont make specific props for feature group, you can also fill in the default options feature group from the leaflet feature group options feature group

    Methods

    after you make a first call, you can call severals method from the instance

    addMoreLine

    this method used to add more line and move the marker, example usage method addMoreLine(latLng, options)

    instance.addMoreLine([-8.822512, 115.186803], {
      duration: 5000, // in milliseconds (optional)
      speed: 25, // in km/h (optional)
      rotateAngle: 141, // (required if rotateMarker enable)
      animatePolyline: true, // (required)
    })
    getMarker

    this is used to get marker instance and will return object class of marker, example usage method getMarker()

    instance.getMarker()
    hideMarker

    this is used to hide/show the marker and this method called from getMarker, example usage method getMarker().hideMarker(boolean)

    instance.getMarker().hideMarker(true)
    activeAnimate (marker)

    this is used to enable/disable the marker and this method called from getMarker, example usage method getMarker().activeAnimate(boolean)

    instance.getMarker().activeAnimate(true)
    activeFollowMarker

    this used to disable/enable screen follow the marker moving and this method called from getMarker, example usage method getMarker().activeFollowMarker(boolean)

    instance.getMarker().activeFollowMarker(true)
    getCurrentPolyline

    this is used to get the polyline instance which is moving, example usage method getCurrentPolyline()

    instance.getCurrentPolyline()
    hidePolylines

    this is used to hide/show the polylines and this method not called from getCurrentPolyline, example usage method hidePolylines(boolean)

    instance.hidePolylines(true)
    activeAnimate (polyline)

    this is used to disable/enable the current polyline which is moving and this method called from getCurrentPolyline, example usage method getCurrentPolyline().activeAnimate(boolean)

    instance.getCurrentPolyline().activeAnimate(true)
    stop

    This is used to end the animation of the marker and polyline that are currently in moving animation and this method not called from getCurrentPolyline, example usage method stop()

    instance.stop()

    Report Issue/Question

    if you have a question about L.MoveMarker or if you found a bug, you can make a issue, more detail how to make a report/question here

    thanks,
    kadek.

    Visit original content creator repository

  • CongressCollector

    What is CongressCollector?

    CongressCollector is a console application that can be used to collect data related to U.S. Congressional Legislative Bills and Resolutions in bulk.

    What are the prerequisites?

    Currently, the application is written to be compiled against the .NET Core SDK / Microsoft.NETCore.App 1.1.0. You can download the current version of the .NET Core SDK here. You can check the project.json file to see all the supported platforms, runtimes, and required dependencies.

    How does it work?

    The U.S. Government Publishing Office (GPO) maintains a database of congressional items such as legislative bills/resolution texts, statuses, and summaries. The easiest way to obtain this data is via the GPO FDsys bulk data site. You can inspect these files manually here. You can read more about the details of the GPO FDsys bulk data at the GitHub repo.

    This application will assemble HTTP requests to retrieve the ZIP files in each container/congress/measure combination from the GPO FDsys system. Each ZIP file contains a collection of XML files representing each legislative item. Additionally, this application will convert the XML files to JSON files to make data consumption less cumbersome.

    GPO FDsys Folder Structure

    In a nutshell, the GPO FDsys system stores XML files for each legislative item in a specific folder structure.

    COLLECTION --> CONGRESS --> MEASURE --> ZIP/XML
    

    COLLECTION is a data collection related to Bill Summaries, Bill Text, or Bill Statuses.

    CONGRESS is the number of the Congress in session at the time of the legislation.

    MEASURE is the legislative measure types such as House Resolutions, Senate Joint Resolutions, and more.

    Some Folder Structure Examples

    BILLSTATUS --> 115TH --> HR --> ZIP/XML
    BILLSTATUS --> 113TH --> SCONRES --> ZIP/XML
    BILLSUM --> 114th --> S --> ZIP/XML
    

    How do I use it?

    First, you’ll need to install the latest .NET Core SDK for Windows or Linux here, depending on your OS.

    Then, you can open a terminal to clone, build, and run the application:

    Clone and Build

    git clone https://github.com/babelshift/CongressCollector.git
    cd ./CongressCollector/src
    dotnet build
    

    Common Uses

    Help Command

    dotnet CongressCollector.dll -?
    
    CongressCollector - For all your congressional data needs.
    
    Usage: CongressCollector [options] [command]
    
    Options:
      -? | -h | --help  Show help information
    
    Commands:
      collect  Initiate the process of collecting bulk data from the GPO FDsys database.
      list     List some valid inputs for commands and options.
    
    Use "CongressCollector [command] --help" for more information about a command.
    

    List Command Help

    dotnet CongressCollector.dll list -?
    
    Usage: CongressCollector list [options] [command]
    
    Options:
      -? | -h | --help  Show help information
    
    Commands:
      collections  List all collections that can be used as an argument of the 'collect' command.
      congresses   List all congresses that can be used in the '-c' option of the 'collect' command.
      measures     List all measures that can be used in the '-m' option of the 'collect' command.
    
    Use "list [command] --help" for more information about a command.
    

    List Collections

    dotnet CongressCollector.dll list collections
    
    'billstatus' - Detailed bill or resolution metadata
    'billsum' - Text summaries of bills, resolutions, or other documents associated with measures such as amendments, reports, or public laws
    

    List Congresses

    dotnet CongressCollector.dll list congresses
    
    '113' - 113th Congress (2013 - 2014)
    '114' - 114th Congress (2015 - 2016)
    '115' - 115th Congress (2017 - 2018)
    

    List Measures

    dotnet CongressCollector.dll list measures
    
    'hconres' - House Concurrent Resolution
    'hjres' - House Joint Resolution
    'hres' - House Simple Resolution
    'hr' - House Bill
    'sconres' - Senate Concurrent Resolution
    'sjres' - Senate Joint Resolution
    'sres' - Senate Simple Resolution
    's' - Senate Bill
    

    Collect Command Help

    dotnet CongressCollector.dll collect -?
    
    Usage: CongressCollector collect [arguments] [options]
    
    Arguments:
      collection  Bulk data collection to retrieve. See 'list collections' for valid inputs.
    
    Options:
      -? | -h | --help            Show help information
      -c | --congress <congress>  Congress for which to receive data. See 'list congresses' for valid inputs.
      -m | --measure <measure>    Legislative measures to retrieve. See 'list measures' for valid inputs.
    

    Collect Specific Legislative Items for a Specific Congress

    House Concurrent Resolutions for the 115th Congress in the BILLSTATUS collection.

    dotnet CongressCollector.dll collect billstatus -c 115 -m hconres
    

    Collect All Legislative Items for a Specific Congress

    All legislative measures for the 115th Congress in the BILLSTATUS collection.

    dotnet CongressCollector.dll collect billstatus -c 115
    

    Collect a Specific Legislative Item for all Congresses

    House Concurrent Resolutions for all congresses in the BILLSTATUS collection.

    dotnet CongressCollector.dll collect billstatus -m hconres
    

    Collect All Legislative Items

    All legislative measures for all congresses.

    dotnet CongressCollector.dll collect billstatus
    

    Output

    The application will output XML and JSON files for each retrieved legislative measure in the following folder structure:

    ./COLLECTION/CONGRESS/MEASURE
    

    Such as:

    ./BILLSTATUS/113/HJRES/BILLSTATUS-113hjres1.json
    ./BILLSTATUS/113/HJRES/BILLSTATUS-113hjres1.xml
    

    Visit original content creator repository

  • plp-mtu-discovery

    plp-mtu-discovery

    Perform Path MTU Discovery without relying on ICMP errors, which are often not delivered.

    This program performs Packetization Layer Path MTU Discovery as described in RFC 4821, which is a more reliable way to detect MTU size in presence of ICMP black holes.

    Rationale

    While TCP connections automatically adjust MTU size over time depending on various indicators (network performance, packet loss, ICMP error messages, …), this is not the case for connection-less protocols.

    When performance is essential, throughput, packet fragmentation and route reliability are three key indicators to analyze in order to optimize stream performance. Since route reliability does not always depend on us, we should strive to maximize throughput while NOT performing packet fragmentation, which can severely degrade performance[1].

    The original proposal for Path MTU Discovery relied on ICMP Fragmentation Needed packets to be delivered when a IPv4 packet with Don’t Fragment field set was too large to be propagated. Unfortunately, some routers do not generate these kind of errors but choose to silently ignore large packets instead. A client has no way to determine the cause of the packet loss.

    A safer approach

    Since all hosts are mandated to support ICMP_ECHO queries, we can exploit the fact that ICMP messages accept an arbitrary amount of data and send different-sized packets to our server. If we turn on the Don’t Fragment field in the IPv4 packet and listen for a response, we are de facto waiting for an ACK (in the form of an ICMP_ECHOREPLY packet) confirming that this MTU size is valid.

    Now we just have to perform a binary search in respect to the size of the packets in order to find the maximum MTU size supported by this route.

    ICMP mode

    When in ICMP mode, some ICMP_ECHO requests of different sizes are generated.

    • If a response from the server is received, that MTU size is considered valid and the threshold is raised.
    • If no response is received after several attempts or some kind of error is received (e.g.ICMP Fragmentation Needed), that MTU size is declared invalid and the threshold is lowered.

    The only requirement of ICMP mode is that the host must be capable to reply to ping messages.

    UDP mode

    The same algorithm applies to UDP packets, but you need to run a server (udp_server.py) on your receiving host in order to send back acknowledgment messages.

    Compiling & Running

    This program should run fine on most Linux distributions and OSX.

    gcc -Wall -Wextra mtu_discovery.c mtu.c -o plpmtu
    

    It should not report warnings/errors. If it does, please open an issue.

    If you want to run in ICMP mode type:

    sudo ./plpmtu -p icmp -s <server-ipaddr>
    

    If you want to run UDP mode instead:

    sudo ./plpmtu -p udp -s <server-ipaddr:port>
    

    Admin rights are required in order to use raw sockets.

    Command line arguments

    Specifier Description
    -p {icmp/udp} Select in which mode to operate.
    -s <addr[:port]> Specify server’s address. If running in UDP mode, you must also specify the destination port by appending ‘:port’ (e.g. -s 8.8.8.8:12345)
    -l <addr:port> Optional. Select on which address to bind(); used in UDP mode; might be removed.
    -t <timeout> Optional. Select the maximum time to wait for a response from the server, default is 1 second; time is expressed in milliseconds.
    -r <max-reqs> Optional. Select the maximum number of failed attempts needed to declare a MTU size invalid, default is 3 attempts.

    Examples

    sudo ./plpmtu -p icmp -s 184.12.26.131
    

    Perform MTU discovery (ICMP mode) with 184.12.26.131.

    sudo ./plpmtu -p udp -s 184.12.26.131:24000 -t 1500 -r 5
    

    Perform MTU discovery (UDP mode) with 184.12.26.131 on port 24000. If a response is not received within 1.5 seconds for 5 times in a row, reduce MTU threshold.

    Docs

    Visit original content creator repository

  • rust-algorithms

    Contest Algorithms in Rust

    Crates.io Version Documentation license Crates.io Downloads Build Status Gitter

    A collection of classic data structures and algorithms, emphasizing usability, beauty and clarity over full generality. As such, this should be viewed not as a blackbox library, but as a whitebox cookbook demonstrating the design and implementation of algorithms. I hope it will be useful to students and educators, as well as fans of algorithmic programming contests.

    This repository is distributed under the MIT License. Contest submissions need not include the license text. Enjoy!

    For Students and Educators

    When learning a new algorithm or data structure, it’s often helpful to see or play with a concrete implementation. As such, this repository catalogues several classic algorithms in their simplest forms.

    In addition, the Rust language has outstanding pedagogical attributes. Its compiler acts as a teacher, enforcing strict discipline while pointing to clearer ways to structure one’s logic.

    For Programming Contests

    The original intent of this project was to build a reference for use in programming contests. As a result, it contains algorithms that are frequently useful to have in one’s toolkit, with an emphasis on code that is concise and easy to modify under time pressure.

    Most competitive programmers rely on C++ for its fast execution time. However, it’s notoriously unsafe, diverting a considerable share of the contestant’s time and attention on mistake prevention and debugging. Java is the next most popular choice, offering a little safety at some expense to speed of coding and execution.

    To my delight, I found that Rust eliminates entire classes of bugs, while reducing visual clutter to make the rest easier to spot. And, it’s fast. There’s a learning curve, to be sure. However, a proficient Rust programmer stands to gain a competitive advantage as well as a more pleasant experience!

    Some contest sites and online judges that support Rust:

    For help in getting started, you may check out some of my past submissions (requires login).

    Programming Language Advocacy

    My other goal is to appeal to developers who feel limited by ancient (yet still mainstream) programming languages, by demonstrating the power of modern techniques.

    Rather than try to persuade you with words, this repository aims to show by example. If you’d like to learn the language, I recommend the official book or Programming Rust.

    Contents

    • Integer index-based adjacency list representation
    • Disjoint set union
    • Euler path and tour
    • Kruskal’s minimum spanning tree
    • Dijkstra’s single-source shortest paths
    • DFS pre-order traversal
    • Connected components
    • Strongly connected components
    • Bridges and 2-edge-connected components
    • Articulation points and 2-vertex-connected components
    • Topological sort
    • 2-SAT solver
    • Dinic’s blocking maximum flow
    • Minimum cut
    • Hopcroft-Karp bipartite matching
    • Minimum cost maximum flow
    • Greatest common divisor
    • Canonical solution to Bezout’s identity
    • Miller’s primality test
    • Fast Fourier transform
    • Number theoretic transform
    • Convolution
    • Rational numbers
    • Complex numbers
    • Linear algebra
    • Safe modular arithmetic
    • Comparator for PartialOrd
    • Binary search: drop-in replacements for C++ lower_bound()/upper_bound()
    • Merge and mergesort
    • Coordinate compression
    • Online convex hull trick (update and query the upper envelope of a set of lines)
    • Statically allocated binary indexed ARQ tree (a.k.a. generic segtree with lazy propagation)
    • Dynamically allocated ARQ tree, optionally sparse and persistent
    • Mo’s algorithm (a.k.a. query square root decomposition)
    • Utility for reading input data ergonomically
    • File and standard I/O examples
    • Generic trie
    • Knuth-Morris-Pratt single-pattern string matching
    • Aho-Corasick multi-pattern string matching
    • Suffix array: O(n log n) construction using counting sort
    • Longest common prefix
    • Manacher’s linear-time palindrome search
    Visit original content creator repository